Cogent confabulation based expert system for segmentation and classification of natural landscape images.
Natural Mediterranean landscape images are becoming more important each day because of the rapidly growing number of installations of automatic wildfire monitoring and surveillance systems [1-3]. Such systems usually consist of three parts: a video camera located on a monitoring location, a transmission system used to transmit data from that monitoring location to the operation center, and an appropriate processing unit with software for automatic image analysis whose main goal is to recognize wildfire smoke or flames on images. New methods for wildfire detection partly rely on knowledge about the type and position of regions on analyzed images, therefore region classification of natural landscape images is an important task in wildfire detection [3-4]. In order to improve the detection algorithms, we have oriented our research efforts to fast and automatic segmentation and classification of regions that may appear on natural Mediterranean landscape images.
In this paper we propose a novel expert system for natural Mediterranean image segmentation and classification based on cogent confabulation theory [5-6] (which is a novel model of cognition) and various knowledge bases. These knowledge bases contain information about local and global features, context of each of the 11 Mediterranean landscape classes that we used, and information regarding optimal color spaces suitable for the classification of those classes. Since this last knowledge base contains information about color probability histograms for 11 Mediterranean landscape classes that we used, and since this is an important part of our expert system, in this paper we also present results of our research related to the search for optimal color spaces for the segmentation of images of natural Mediterranean landscape. Nine different color spaces (RGB, HSV, HLS, Ohta, CIE-XYZ, CIE L*a*b*, CIE L*u*v*, YCgCr and YCbCr) were used in this research, and our aim was to discover the optimal ones for the classification of different semantic classes that can usually be found in images of Mediterranean landscape. Since color helps to discriminate between many types of classes, it is still one of the most commonly used feature in automatic image classification systems. Because of this, our particular effort was on research about optimal color spaces for the classification of regions found on natural Mediterranean landscape images. For example, in static images the sea will usually appear blue and the vegetation will usually appear green, so just by using color alone the difference between these two classes can be easily observed. However, since many different color spaces exist it is sometimes difficult to select the one that would be optimal for the image processing task at hand. A color space that is suitable for one set of images is often not suitable for another, and this can lead to many difficulties. Although many papers on evaluation of different color spaces exist (e.g. for edge classification in outdoor scenes , skin detection in face recognition systems , etc.), as far as we know an evaluation of different color spaces for use in the classification of natural Mediterranean landscape images does not exist, and that is the reason we decided to present it in this paper.
The expert system we propose in this paper was evaluated on a FESB MLID dataset and compared against Normal Bayes classifier and Multi-Layer Perceptrons (MLP). The results that we obtained indicate that the proposed expert system is faster and more accurate on the testing part of the FESB MLID dataset than the classifiers it was compared against, and is therefore suitable for natural landscape classification and segmentation.
This paper is structured as follows. In Section 2 we give an overview of the related work. In Section 3 we propose an expert system for the automatic segmentation and classification of Mediterranean landscape images, which we will refer to as the cogent confabulation based (CCB) expert system. As this expert system is based on various knowledge bases that contain information about color probability histograms and optimal color spaces for color segmentation, in Section 4 we present our research about optimal color spaces for the segmentation and classification of regions that can commonly be found in Mediterranean landscape images. Section 5 gives details of the segmentation and classification procedures used in the proposed expert system. Section 6 presents experimental results (results of the comparison of the proposed expert system with the standard classifiers), and it also presents the FESB Mediterranean Landscape Image Dataset (FESB MLID) used in this research. FESB MLID dataset consists of 400 images of natural Mediterranean landscape and their corresponding hand-labeled segmentations. Finally Section 7 gives a conclusion and discusses future work.
II. RELATED WORK
This section of the paper gives an overview of the related work, and is split into 3 sections: natural landscape classification and segmentation, color spaces and cogent confabulation.
A. Natural landscape classification and segmentation
Selection of a proper color spaces for different computer vision tasks has been studied in scientific literature, but color space evaluation and/or comparison intended for purposes of classification, segmentation, detection and/or recognition of classes that can be found in natural landscape images (e.g. classes such as water or sky) is still rare. On the contrary, this kind of evaluation and/or comparison is abundant for other purposes such as face detection. For example, Chaves-Gonzalez et al. in  presented a study of color spaces for skin detection in face recognition systems. They compared ten different color spaces (RGB, CMY, YUV, YIQ, YPbPr, YCbCr, YCgCr, YDbDr, HSV or HSI and CIE-XYZ) and found that the most appropriate ones for skin color detection are HSV, YCgCr and YDbDr. Out of those three models, HSV was deemed the winner in their study. Wang et al.  used color space fusion and dimension reduction scheme for vehicle color classification. Selection and fusion of color models was also used by Stokman and Gevers in  for image feature detection.
When it comes to natural outdoor scenes, we found that the following papers were most similar to our own work: [11-13].
Marti et al.  suggested a top-down approach for outdoor scene description in which they selected for each object class a subset of color and texture features that best characterised it, i.e. that had the potential to best distinguish it from the other object classes. They defined an object class as a real object in specific outdoor conditions. The approach that they proposed allows the incorporation of new objects in order to better describe a scene. The objects that they used in their paper are: road, leaves, sky and ground.
Bosch et al.  suggested an approach for natural outdoor scenes segmentation and description in which the subset of color and texture features was selected for each object with the goal of best distinguishing that object from the remaining ones. They used Sequential Forward Floating Search algorithm for feature selection for each object. They specified which features they used for each object in different image datasets that they used. They evaluated their method on three different datasets. On the first two datasets they used the same set of classes (sky, grass, road, vegetation and ground--everything else they considered to be unknown), and a different set of classes on the third dataset (rhino/hippo, polar bear, vegetation, sky, water, snow and ground). The system that they proposed allows the learning of new categories given the training examples of those categories. To compensate for objects in various weather conditions, each object was modeled as an object class.
Celen and Demirci  performed fire detection in seven different color models, and found that for the detection of fire pixels the best color models (out of the ones that they compared) are CIE L*a*b* and YCrCb. They also searched for the best color model for smoke, and selected best features for the detection of fire and smoke. The process of construction of color probability histograms that we used in this paper is the same as the process of histogram construction used in , although we did not group similar likelihood values and store them in lookup table as the authors in  did. Additionally, one of the steps that the authors in  used while deciding whether a pixel could represent fire or not was computing its likelihood of belonging to fire. They accomplished this by multiplying likelihoods (obtained through the lookup table) of the intensity values belonging to pixel's channels in a given color space. We also used this method while performing image classification (only instead of a lookup table we used color probability histograms).
Additionally, Qelik et al.  also used different color models for the detection of fire and smoke.
In the work presented in this paper we did not use all of the same classes, features and color spaces as the authors in the aforementioned papers did. We focused on a complete segmentation and classification of images of natural landscape, i.e. we took into account all of the classes that we believe occur most frequently in images of natural Mediterranean landscape. Additionally, since we found no other published research on natural Mediterranean landscape segmentation and classification we had to compare the results of our proposed expert system to well-known classifiers instead.
B. Color spaces
Since one of the major contributions of this paper is the analysis revolving around the search for optimal color spaces for the segmentation of images of natural Mediterranean landscape, in this section of the paper we give a list of color spaces that we used and some details on their implementation. The well-known color spaces that we decided to use in this paper are: RGB, HSV, CIE L*a*b*, CIE L*u*v*, YCgCr (proposed in ), YCbCr, HLS, CIEXYZ and the color space proposed by Ohta et al. in . (We will refer to the color space proposed by Ohta et al. in , and defined with color features I1, I2' and I3', as the Ohta color space.) We used OpenCV (Open Source Computer Vision, http://opencv.org/) functions to convert input images into the following color spaces: YCbCr (implemented in OpenCV as YCrCb), HSV, HLS, CIE L*a*b* and CIE L*u*v*. The input to these functions were 8-bit RGB images that were assumed to be linear in the case of a conversion to CIE L*a*b*. Functions for converting RGB image into YCgCr and CIE-XYZ color spaces we implemented ourselves by using the directions (with some of our own minor alterations) and information about those color spaces that are provided in . We also implemented the function for converting RGB image into Ohta color space by using the information about it given in  (with some of our own minor alterations).
C. Cogent confabulation
Cogent confabulation [5-6] is a novel model of cognition proposed by Robert Hecht-Nielsen in 2005. In cogent confabulation theory the overall likelihood that a pixel P, having x, y and z as red, green and blue components, belongs to a certain class c is a conditional probability p(xyz|c), named as a cogency , contrary to Bayesian decision theory where it is expressed as a posterior probability p(c|xyz). Cogency is the probability of the assumed facts xyz being true, given an assumption that the event c is true, while Bayesian reasoning suggests to chose the event c that has the highest posterior probability p(c|xyz) giving the assumed facts xyz (for detail discussion about cogency please refer to [5-6, 17]). Beside the fact that according to [5-6] cogent confabulation based reasoning better represents the human model of reasoning, its main advantage are simple calculations. Cogency could be approximated according to the cogent confabulation theory by a simple confabulation product, as shown in equation (1) [5-6].
p [(xyz|c).sup.3] [approximately equal to] C x p(x|c) x p(y|c) x p(z|c) (1)
In some cases, C might function approximately as a positive constant . Hecht-Nielsen  suggests that maximizing the confabulation (product of simple conditional probabilities: p(x|c) x p(y|c) x p(z|c)) will also maximize cogency (conditional probability p(xyz|c)). In  we have proposed that the process of cogency estimation by maximizing confabulation product can be used as a general effective classification tool for wildfire and natural risk observers, and therefore we have applied it also in optimal color spaces evaluation, and also in the proposed CCB expert system classifier. The similar mechanism was proposed in  for the classification of fire and smoke, although it was not referred to as the result of the cogent confabulation theory.
Maximization of confabulation can also look similar to the maximization a posteriori of a Naive Bayes classifier, but it is not the same as Naive Bayes classifier because it does not require the probability that the class will occur (p(c)) and the probability that the variable will occur (e.g. p(x)).
III. EXPERT SYSTEM FOR SEGMENTATION AND CLASSIFICATION OF LANDSCAPE IMAGES
Our main task was to design and construct an expert system for automatic segmentation of Mediterranean landscape images in 11 semantic classes. Since we decided to work with natural Mediterranean landscape images typically extracted from wildfire surveillance cameras video stream, we needed to determine which classes those images usually contain in order to be able to automatically classify them. We have decided to work with 11 classes presented in  (although in  they were called categories): smoke, clouds and fog, sun and light effects, sky, water surfaces (e.g. sea, lakes, rivers, etc.; in  this class was called sea), distant landscape, stone, distant vegetation, close vegetation, low vegetation and agricultural areas, and buildings and artificial objects. We should note that these 11 classes were selected primarily for the purpose of wildfire smoke detection , but we still found them to be useful since they covered all of the classes that can usually be found in images of Mediterranean landscape. However, some of these classes tend to be similar in appearance in certain contexts, so that makes automatic Mediterranean landscape image classification very difficult. In addition to these 11 classes, we also added a class unknown areas, so in total 12 classes were defined. We should point out that the class unknown areas is not used in classification process itself, but rather only in the evaluation of the classification process.
The proposed expert system fuses information about optimal color spaces for each class, local and global image features, and context information extracted from the train set of images. One of our important tasks was to make it fast and suitable for real time applications, so therefore the proposed expert system is based on a simple classifier derived from the cogent confabulation theory [5-6]. This classifier was used in the teaching procedure, during the construction of knowledge bases, but also as a final classifier in the segmentation and classification of unknown images. The proposed cogent confabulation based (CCB) expert system was compared against two well-known classifiers (Multi-Layer perceptrons and Normal Bayes), and the obtained results presented in this paper indicate that both the accuracy of the proposed expert system and its execution time outperform the other classifiers that it was compared against.
Behind every expert system there is a knowledge base that contains expert information about the task at hand. The knowledge base behind the proposed CCB expert system consists of three parts: (1) Color probability histograms knowledge base (CPH-KB), (2) Global color histograms knowledge base (GCH-KB), and (3) Contextual information knowledge base (CI-KB).
Color probability histograms knowledge base (CPH-KB) includes conditional probabilities that a certain color value x from the appropriate color channel could appear in certain Mediterranean landscape class c - p(x|c). This knowledge base also includes the knowledge about optimal color spaces where classification of each class in Mediterranean landscape images gives the best results. Based on this research, described in details in section 4, in the proposed CCB expert system we have only used color probability histograms related to color channels in RGB, HSV, HLS, CIE-XYZ, CIE L*a*b*, YCbCr and YCgCr color spaces. We did not use color probability histograms related to Ohta or CIE L*u*v* color spaces, because neither of those two color spaces was deemed optimal for the classification of any of the proposed classes.
Global color histograms knowledge base (GCH-KB) are used for scene classification. While deciding which global image features should we include in our knowledge database, we tested 29 of them and selected the top three. Those global features are global color histograms in the following color channels: 3rd channel of the CIE-XYZ color space, 1st channel of the HSV color space, and 1st channel of the HLS color space. For each image in the train set of images from the FESB MLID dataset we have calculated three global color histograms, one for each of those three channels, and labeled them with a vector containing information about the percentage of representation of each class in that image. This information is important for the proposed CCB expert system, because for every new image that we want to classify we calculate its three global color histograms, compare them to the corresponding histograms in the knowledge database, find the ones that are closest to them, and obtain their labels. These labels give us information about the global scene classification, i.e. they tell us which classes are likely to appear in our image and how much area in the image are they likely to occupy.
Contextual information knowledge base (CI-KB) was used to improve the overall classification performance. For contextual information we have calculated the average probabilities of each class that show how likely it is that a particular class will appear in the top, middle, or bottom part of the image (this method was also used in ). We have calculated these probabilities on train images from the FESB MLID dataset, stored them in a knowledge base, and used them in the cogent confabulation pixel-based classification procedure explained in Section 5.
Since CPH-KB knowledge base is essential for the proposed CCB expert system, in the next section we will describe in details our research related to its construction.
IV. THE SELECTION OF OPTIMAL COLOR SPACES IN SEGMENTATION AND CLASSIFICATION OF NATURAL LANDSCAPE IMAGES
In this section of the paper we present an evaluation of the use of 9 different color spaces in the classification of 11 classes that can usually be found in the images of Mediterranean landscape. We discuss the color-based classification process and present the optimal color spaces for each class according to different criteria.
A. Cogent confabulation color-based classification
From the 200 train images from the FESB MLID dataset we calculated color probability histograms that tell us, for each class and for each color channel of particular color space, how likely it is that a certain value of the color channel will appear for a certain class. Fig. 1 shows examples of color probability histograms for two classes (close vegetation and smoke) calculated for the blue channel of the RGB color space.
The procedure of the color-based classification that we used is based on cogent confabulation theory [5-6] and detailed in the following text. Let us suppose that we want to classify one image pixel P(x,y,z) from one test image from the FESB MLID dataset in the RGB color space, so x, y and z are red, green and blue values of that pixel. The first step is to determine, from color probability histograms, how likely is it that x will appear in the red channel of all 11 classes. The result of this step are 11 conditional probabilities, named p(x|c), one for each class c. The same procedure is used for the other two channels, so p(y|c) and p(z|c) give information about how likely is it that value y will appear in the green channel and z in the blue channel of all 11 classes (determined with the variable c), respectively.
Finally, for each pixel P(x,y,z) we obtain 33 conditional probabilities (p(x|c), p(y|c), p(z|c), where c corresponds to classes from 1 to 11). These information are then stored in the color probability histogram knowledge base (CPH-KB).
The winner of the classification procedure that indicates to which class pixel P(x,y,z) most likely belongs to is the class c that has maximal confabulation product, as suggested by the cogent confabulation theory [5-6]. The procedure is repeated for all test image pixels, as well as for all of the eight remaining color spaces, as shown in Fig. 2. At the end we obtain nine labeled images, one for each of the color spaces that we used. An example of these images is given in Fig. 3 and Fig. 4, for one image from the FESB MLID dataset.
The reason that we have decided to use this simple classification technique on image pixel level was in order to put more emphasis on the selection of the optimal color space for each class rather than on the selection of the optimal classifier, optimal image segmentation technique or optimal scale on which to perform our analysis. But as Section 6 shows, cogent confabulation based classifier outperformed other well-known classifiers that it was compared against, and was much faster. Therefore it is more suitable for real-time applications and that was one of our research goals.
B. The selection of optimal color spaces
Fig. 5 shows the overview of the procedure that we used for determining the optimal color spaces for each of the 11 Mediterranean landscape classes that we have used.
We have applied cogent confabulation color-based classification algorithm to 200 test images from the FESB MLID test dataset, and for each image we obtained 9 segmented and classified images, one for each color space that we used. We compared these images to their corresponding ground truth (GT) images to discover how accurately were they automatically classified. We presented the obtained results with confusion matrices, and for the purposes of this paper we used two types of confusion matrices. The first type shows precision values for a given class, while the second one shows recall values. The confusion matrix that shows precision values is obtained by dividing each column element of the "raw" confusion matrix (the one that is not yet normalized, i.e. the one with the actual pixel counts) with the sum of all of the elements belonging to that column (including the pixels that the classification algorithm failed to classify). The confusion matrix that shows recall values is obtained by dividing each row element of the "raw" confusion matrix with the sum of all of the elements belonging to that row (including the pixels that the classification algorithm failed to classify). We can see that precision gives us information about how many pixels classified as belonging to a certain class actually belong to that class, while recall gives us information about how many pixels that belong to a certain class are actually classified at belonging to that class. In this paper, our confusion matrices have an additional column (the last column) that contains information about the pixels that the color-based classification algorithm failed to classify. Those are the pixels whose label could not be determined because the probabilities of it occurring in each color space and for each class that was used were zero. We ignored pixels whose GT classification was unknown.
In this paper we give visual representations of the confusion matrices that we obtained, instead of the confusion matrices themselves (since there would be too many of them). We assumed that this would prove to be more effective and intuitive, since it will allow the readers to get a quick overview of the precision and recall values without the need to examine 18 different confusion matrices. Visual representations of the confusion matrices that represent precision values are shown in Fig. 6, while the ones that represent recall values are shown in Fig. 7. Darker blocks in those images correspond to higher percentages in the confusion matrices. The names of the classes in Fig. 6 and Fig. 7 are given by using the following convention: (1) Smoke, (2) Clouds and fog, (3) Sun and light effects, (4) Sky, (5) Water surfaces, (6) Distant landscape, (7) Stone, (8) Distant vegetation, (9) Close vegetation, (10) Low vegetation and agricultural areas, (11) Buildings and artificial objects, and (12) Unknown.
It can be seen from Fig. 7 that there is some confusion between certain classes. Smoke, clouds and fog, water surfaces, distant landscape, and stone can all have very similar color features under certain conditions and certain distance from the camera, and therefore can often be confused with one another. Other classes that are often confused with one another are distant vegetation, close vegetation and low vegetation and agricultural areas, because these classes mostly consist of different shades of green and brown colors, and it is very difficult to determine the correct class if an image is classified on a pixel level. Buildings and artificial objects also proved to be difficult to classify, given the broad range of colors that the pixels belonging to that class can assume. From this we conclude that color information alone does not have enough power to separate certain classes from one another, so additional information (e.g. contextual) should be used alongside it. However, we also conclude that there are color spaces that are better at separating certain natural landscape classes than the others, so they should provide a better starting point for the classification of those classes than some randomly selected color space.
In order to conclude which color space should be used for the classification of different Mediterranean landscape classes, we calculated F-measures from the confusion matrices. F-measures are used to combine precision and recall results and give a single measure that can estimate the classification quality. The well-known equation that is used for calculating the F-measure is equation (2) .
F - measure = ([[beta].sup.2] + 1) x precision x recall/[[beta].sup.2] x precision + recall (2)
Different types of F-measures exist. In [F.sub.1]-measure, the [beta] parameter is 1 so the equal weight is given to both recall and precision. In [F.sub.2]-measure, the [beta] parameter is 2 and more weight is given to recall. In [F.sub.0.5]-measure, the [beta] parameter is 0.5 and more weight is given to precision.
Table I presents a summary of optimal color spaces, given different criteria, for specific Mediterranean landscape classes and uses the same terminology for class names as Fig. 6 and Fig. 7 do. Additionally, it also gives the percentages that show how accurate was a given measure in the given optimal color space. The results presented in this table were obtained for the 200 test images from the FESB MLID dataset. It can be seen from this table that the optimal color spaces for most Mediterranean landscape classes will change with different criteria, meaning that for most classes the optimal color space that would maximize precision, recall, and different F-measures does not exist. When choosing the optimal color space for color-based Mediterranean landscape image classification, we suggest the focus should be on the criteria that is most important to the task, so that the optimal color space for that particular task could be selected accordingly. For example, in wildfire smoke detection high recall is preferred over high precision since it is of extreme importance to detect every wildfire smoke that occurs so that the alarm could be raised and firefighters alarmed. In this case it would be better to deal with a higher number of false smoke alarms than to miss any actual smoke. For cases where multiple color spaces cannot be used in Mediterranean landscape image classification, we wanted to discover which single color space would be optimal to use.
Table II shows an overview of Mediterranean landscape image classification results if that classification is performed in one color space only. These results are calculated for 200 test images from the FESB MLID dataset. It can be seen from the table that the overall optimal color space for Mediterranean landscape image classification is HSV, and the weakest ones are RGB and CIE-XYZ.
C. Contextual information knowledge base
For contextual information we have calculated the average probabilities of each class that show how likely it is that a particular class will appear in the top, middle, or bottom part of the image on 200 train images from the FESB MLID dataset. Results are presented in Table III.
V. EXPERT SYSTEM SEGMENTATION AND CLASSIFICATION PROCEDURE
In this section of the paper we explain in detail each step of the CCB expert system for the segmentation and classification of landscape images. Proposed algorithm concurrently performs both image segmentation and regions classification. It consists of 7 steps schematically shown in Fig. 8, and detailed below.
Step 1. Calculation of global color descriptors of the input image, and estimation of its closest neighbors in the global color histogram knowledge base (GCH-KB) according to the L 1-norm. This step gives us an estimate of the classes that can be found in the input image, as well as an estimate of their areas. Information are stored and used in step 4.
Step 2. Pixel-based classification of input image based on maximization of cogency (xyzq|c). Values x, y and z are the pixel's intensity values in 3 color channels, and q is the probability that shows how likely it is that class c can be found in the current part of the image (Table III). This step is performed for 7 color spaces, and its end result are 7 classified images.
Step 3. If the pixel is classified as the same class in all 7 color spaces in the previous step, than that class becomes its final label.
Step 4. If there are any more unclassified pixels in the image, they are classified as follows. The information about optimal color spaces from Table I according to the F1 measure is used (we use the [F.sub.1]-measure because it gives equal weight to both precision and recall), and it is taken into account how successfully were classes classified in their designated optimal color spaces. The higher their [F.sub.1]-measure is, the higher weight in the classification process will they have. For example, [F.sub.1]-measure for the class 3 Sun and light effects in its optimal color space (RGB) was 82.843%, while the [F.sub.1]-measure for the class 4--Sky in its optimal color space (HLS) was 54.406%. In our proposed system this means that if the pixel is classified in the RGB color space as Sun and light effects and in the HLS as Sky, the first one Sun and light effects will take precedence in the classification system because it is more likely that this classification is accurate. The list all of the classes in the descending order according to their [F.sub.1]-measure could be derived from Table I: 1. Sun and light effects, 2. Close vegetation, 3. Sky, 4. Clouds and fog, 5. Low vegetation and agricultural areas, 6. Distant vegetation, 7. Distant landscape, 8. Stone, 9. Buildings and artificial objects, 10. Water surfaces, and 11. Smoke. In addition to the weight of the Fj -measures, the pixel will only be classified as a particular class if that class is possible for that input image, according to the comparison of input image with global color histogram knowledge base (GCH-KB) calculated in step 1.
Step 5. If there are any more unclassified pixels in the image, they are classified similarly to the previous step. The only difference is that global image labels are not taken into account in this step of the algorithm.
Step 6. If there are any more unclassified pixels in the image, we classify them by maximizing cogency p(xyz|c) and in this case x, y and z are the pixel's intensity values in 3 color channels of the HSV color space. We chose to use this color space because in Table II it has the highest [F.sub.1]-measure for Mediterranean landscape image classification performed in a single color space.
Step 7. If some pixels still remain unclassified they are labeled as class 12--Unknown areas.
Table IV shows how the [F.sub.1]-measure and the percentage of unclassified pixels changes with the addition of different classification steps to the proposed CCB expert system. The steps that we focused on were the steps 3-6 detailed above, since they are the main classification steps. Even though the final [F.sub.1]-measure was a little bit lower after step 6 than it was after step 4, we still take that lower [F.sub.1]-measure as the final one since the number of unclassified pixels is much lower after step 6 than it was after step 3.
VI. EXPERIMENTAL RESULTS
In this section of the paper we give details about the FESB MLID dataset that we used in this paper, and we compare the proposed expert system with two well-known classifiers--Normal Bayes and MLP.
A. Mediterranean Landscape Image Dataset
Mediterranean Landscape Image Dataset (FESB MLID, http://wildfire.fesb.hr/) is a set of 400 natural Mediterranean landscape images along with their ground truth (GT) segmentations. 12 classes that we listed in the previous text were used in the process of manual segmentation of the images from this dataset. The name of the dataset is a combination of two acronyms, the first one being "FESB" which is a Croatian acronym for the Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, and the second one being "MLID" which stands for "Mediterranean Landscape Image Dataset". Images of Mediterranean landscape from the FESB MLID dataset were obtained mostly through the Croatian iForestFire (Intelligent Forest Fire Monitoring System)  project. Some of the other images used in the FESB MLID dataset came from  and from our departmental internal image and video database (http://wildfire.fesb.hr/). In its initial stages, FESB MLID dataset was used for model validation in .
Out of the 400 images currently in the FESB MLID dataset, the first 200 are used as train images and the second 200 as test images in this paper. We should note that although FESB MLID dataset consists of a large number of images, it will expand in the future as we acquire and segment new images of the Mediterranean landscape. Fig. 9 shows an example image from the FESB MLID dataset and its GT segmentation, and Fig. 10 shows examples of class labels.
B. Comparison with the other classifiers
In this section we compare the proposed CCB expert system with different well-known classifiers: Multi-Layer Perceptrons (MLP) and Normal Bayes. The results that we obtained indicate that the accuracy of the CCB expert system is more accurate than that of the mentioned classifiers, and its execution time is much lower.
Additionally, we compared the proposed CCB expert system to the K-Nearest Neighbours (K-NN) and Support Vector Machines (SVM) classifiers. We found these classifiers to be problematic in our case because of the computer memory and speed complications that occurred when we tried to classify 200 test images from the FESB MLID dataset with them. We encountered computer memory issues while trying to create a file that would contain feature vectors calculated for the train images from the FESB MLID dataset. There were far too many of these feature vectors and they caused insufficient memory errors. We solved this problem for Normal Bayes and MLP classifiers by saving these feature vectors into multiple smaller files instead of one large one, and then training the classifiers on the first file and updating them with the rest of the files. Unfortunately, this approach could not be used for the SVM classifier as it does not support incremental learning, and that is why we could not compare this classifier to the proposed CCB expert system. Additionally, we also decided against comparing the K-NN classifier to the proposed CCB expert system. This was decided because the K-NN classification was extremely slow as it took over 26 minutes to classify a single image from the FESB MLID dataset, even though the classification was being performed on non-overlapping blocks of 5x5 pixels instead of pixels. From this it can be concluded that once performed on pixels, the K-NN classifier would be extremely slow and not useful for real-time applications that require pixel-based classification.
Finally, we trained the MLP and Normal Bayes classifiers on 200 train images from the FESB MLID dataset. Each pixel from those images that was manually segmented into one of the 11 Mediterranean landscape classes that we used was described by a feature vector consisting of 27 average intensity values (one average intensity per color channel from 9 color spaces that we used), and a value describing its position in the image (top, middle or bottom). Each of these vectors had a label assigned to it that specified to which class it actually belonged to. These vectors were used by the MLP and Normal Bayes classifiers as a knowledge base in their testing stage, i.e. the classifiers used them in order to predict classes of pixels in images from the test part of the FESB MLID dataset.
In the validation process of the proposed CCB expert system we used confusion matrices that showed precision and recall results for the classification of each class. From these confusion matrices we calculated more representative [F.sub.1]-measure for each class (and for each classifier), and finally calculated the average [F.sub.1]-measure for each class (and for each classifier). Table V and Fig. 11 show average classification results for the MLP classifier, Normal Bayes classifier, and the proposed CCB expert system for 200 test images from the FESB MLID dataset. It can be seen from the table that the proposed system significantly outperforms both the MLP and the Normal Bayes classifiers, as its [F.sub.1]-measure is higher and its execution time lower.
In the last few years there has been a steady increase in the number of automatic, image-processing based wildfire monitoring and surveillance systems. Places that are commonly most affected by wildfires are areas with hot and dry summers, such as the coastal areas around the Mediterranean sea. With this in mind, we focused our research on segmentation and classification of natural Mediterranean images, in the hopes that this research will help in the future automatic detection of wildfires and their prevention.
In this paper we aimed to discover which color spaces, out of nine different ones that we have tested, are most suitable for color-based classification of classes that can usually be found in images of natural Mediterranean landscape. While we found that there are color spaces that are better than others for segmentation and classification of different natural landscape classes, we concluded that to really be able to automatically differentiate between various classes, color information alone is not enough. We therefore proposed a cogent confabulation based expert system for segmentation and classification of natural landscape images that fuses information about the context of each class, optimal color spaces, and local and global image features. The results that we obtained indicate that the accuracy of the proposed expert system is higher and its' execution time lower than those of various well-known classifiers. Since natural landscape image segmentation and classification is often used in real-time applications, this would make the proposed expert system effective and efficient to use. Additionally, we presented a FESB MLID dataset on which we conducted our research. FESB MLID dataset consists of 400 images of natural landscape, alongside their hand labeled segmentations. It is freely available online to anyone doing research in vision and image processing. In our future work we plan to focus on expanding the FESB MLID dataset with new images of natural Mediterranean landscape, and on adding additional classes (such as fire and snow) to the list of Mediterranean landscape classes.
Even though this paper presents an expert system for the automatic segmentation and classification of regions on typical Mediterranean landscape images, the same procedure could be used for any other landscape images, as well as for any other definition of classes, but appropriate knowledge bases have to be created beforehand.
This work was supported in part by the Ministry of Science, Education and Sport of the Republic of Croatia under grant "Intelligent observers in real and virtual environment".
The original work that was presented in this paper was part of Maja Braovic's doctoral dissertation , and in smaller part was published as an abstract , and as a poster , and is used here with permission. The authors would like to express their gratitude to everyone who contributed to the creation of the FESB MLID dataset, and especially to Marin Bugaric, Toni Jakovcevic and Maja Stula who obtained some of the images that are included in the dataset, and to the students from the University of Split, Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture that helped to hand-label some of the images in the initial stages of the FESB MLID image database development. Finally, we would like to thank the anonymous reviewers who helped to improve this paper.
Digital Object Identifier 10.4316/AECE.2017.02012
 M. Bugaric, T. Jakovcevic, D. Stipanicev, "Adaptive estimation of visual smoke detection parameters based on spatial data and fire risk index", Computer Vision and Image Understanding, vol. 118, pp. 184-196, 2014. doi:10.1016/j.cviu.2013.10.003
 D. Krstinic, T. Jakovcevic, D. Stipanicev, "Histogram-based smoke segmentation in forest fire detection system", Information Technology and Control, vol. 38, no. 3, pp. 237-244, 2009.
 T. Jakovcevic, D. Stipanicev, D. Krstinic, "Visual spatial-context based wildfire smoke sensor", Machine Vision and Applications, vol. 24, issue 4, pp. 707-719, May 2013. doi:10.1007/s00138-012-0481-x
 T. Jakovcevic, "Wildfire-smoke detection based on visible-spectrum image analysis" (In Croatian: "Detekcija dima pozara raslinja analizom slika dobivenih u vidljivom dijelu spektra"), doctoral dissertation, Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Croatia, 2011.
 R. Hecht-Nielsen, "Cogent Confabulation", Neural Networks, vol. 18, no. 2, pp. 111-115, March 2005. Doi:10.1016/j.neunet.2004.11.003
 R. Hecht-Nielsen, "The Mechanism of Thought", International Joint Conference on Neural Networks, Vancouver, Canada, pp. 419-426, July, 16-21 2006. doi:10.1109/IJCNN.2006.246712
 E. A. Khan, E. Reinhard, "Evaluation of color spaces for edge classification in outdoor scenes", IEEE International Conference on Image Processing, vol. 3, pp. 952-5, 2005. doi:10.1109/ICIP. 2005.1530551
 J. M. Chaves-Gonzalez, M. A. Vega-Rodriguez, J. A. Gomez-Pulido, J.M. Sanchez-Perez, "Detecting skin in face recognition systems: A colour spaces study", Digital Signal Processing, vol. 20, issue 3, pp. 806-823, 2010. doi:10.1016/j.dsp.2009.10.008
 Y.-C. Wang, C.-C. Han, C.-T. Hsieh, K.-C. Fan, "Vehicle color classification using manifold learning methods from urban surveillance videos", EURASIP Journal on Image and Video Processing, 2014. doi:10.1186/1687-5281-2014-48
 H. Stokman, T. Gevers, "Selection and fusion of color models for image feature detection", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, pp. 371--381, 2007. doi:10.1109/TPAMI.2007.58
 A. Bosch, X. Munoz, J. Freixenet, "Segmentation and description of natural outdoor scenes", Image and Vision Computing, vol. 25, issue 5, pp. 727-740, 2007. doi:10.1016/j.imavis.2006.05.015
 J. Marti, J. Freixenet, J. Batlle, A. Casals, "A new approach to outdoor scene description based on learning and top-down segmentation", Image and Vision Computing, vol. 19, issue 14, pp. 1041-1055, 2001. doi:10.1016/S0262-8856(01)00065-8
 V. Burak Celen, M. Fatih Demirci, "Fire detection in different color models", Proceedings of the 2012 International Conference on Image Processing, Computer Vision, & Pattern Recognition, 2012.
 T. Qelik, H. Ozkaramanli, H. Demirel, "Fire and smoke detection without sensors: image processing based approach", 15th European Signal Processing Conference (EUSIPCO 2007), pp. 1794-1798, 2007.
 J. J. de Dios, N. Garcia, "Face detection based on a new color space YCgCr", International Conference on Image Processing, pp. III-909III-912, 2003. doi:10.1109/ICIP.2003.1247393
 Y.-I. Ohta, T. Kanade, T. Sakai, "Color information for region segmentation", Computer Graphics and Image Processing, vol. 13, pp. 222-241, 1980.
 D. Stipanicev, Lj. Seric, M. Braovic, D. Krstinic, T. Jakovcevic, M. Stula, M. Bugaric, J. Maras, "Vision based wildfire and natural risk observers", 2012 3rd International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 37-42, 15-18 October 2012. doi:10.1109/IPTA.2012.6469518
 M. Sokolova, N. Japkowicz, S. Szpakowicz, "Beyond accuracy, F score and ROC: a family of discriminant measures for performance evaluation", AI 2006: Advances in Artificial Intelligence: 19th Australian Joint Conference on Artificial Intelligence. Lecture Notes in Computer Science, vol. 4304, pp. 1015-1021, 2006. doi:10.1007/11941439_114
 D. Stipanicev, "Intelligent forest fire monitoring system--from idea to realization", Annual 2010/2011 of the Croatian Academy of Engineering, pp. 58--73, 2012.
 T. Roncevic, M. Braovic, D. Stipanicev, "Non-parametric context-based object classification in images", Information Technology and Control. vol. 46, no. 1, pp. 86-99, 2017. doi:10.5755/j01.itc.46.1.13610
 M. Braovic, "Segmentation and classification of non-transparent and semi-transparent regions on natural landscape images" (In Croatian: Segmentacija i klasifikacija neprozirnih i poluprozirnih regija na slikama prirodnog krajolika), doctoral dissertation, Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Croatia, 2015.
 M. Braovic, "Color based region classification in Mediterranean landscape images", Abstract Book--Fourth Croatian Computer Vision Workshop/Editors: Sven Loncaric and Josip Krapac, Zagreb, 2015.
 M. Braovic, Color-based region classification in Mediterranean landscape images", The 2nd ACROSS Workshop on Advanced Cooperative Systems, Poster Session, Zagreb, Croatia, 2016.
Maja Braovic, Darko Stipanicev, Damir Krstinic
Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Rudera Boskovica 32, 21000 Split email@example.com
Caption: Figure 1. Examples of color probability histograms for two predefined classes. The histograms are obtained for the blue channel of the RGB color space
Caption: Figure 2. Cogent confabulation based classification and segmentation of images in different color spaces
Caption: Figure 3. RGB representations of different color spaces. First row: RGB, CIE-XYZ, Ohta, HSV, HLS. Second row: CIE L*a*b*, CIE L*u*v*, YCgCr, YCbCr
Caption: Figure 4. Cogent confabulation based segmentation and classification performed in different color spaces. First row: RGB, CIE-XYZ, Ohta, HSV, HLS. Second row: CIE L*a*b*, CIE L*u*v*, YCgCr, YCbCr (image best viewed in color)
Caption: Figure 5. Overview of the procedure for optimal color spaces selection
Caption: Figure 6. Visual representation of the precision matrices. First row shows visual representations of the confusion matrices for the RGB, CIE-XYZ and Ohta color spaces, the second row for the HSV, HLS and CIE L*a*b* color spaces, and the third row for CIE L*u*v*, YCgCr and YCbCr color spaces respectively. Darker blocks correspond to higher percentages
Caption: Figure 7. Visual representation of the recall matrices. First row shows visual representations of the confusion matrices for the RGB, CIE-XYZ and Ohta color spaces, the second row for the HSV, HLS and CIE L*a*b* color spaces, and the third row for CIE L*u*v*, YCgCr and YCbCr color spaces respectively. Darker blocks correspond to higher percentages
Caption: Figure 8. Proposed cogent-confabulation based expert system for segmentation and classification of Mediterranean landscape images
Caption: Figure 9. Example image from the FESB MLID dataset and its corresponding GT segmentation. Different shades of gray on the GT image correspond to different classes
Caption: Figure 10. Examples of different class labels
Caption: Figure 11. Comparison of the performance of the proposed CCB expert system with the MLP and Normal Bayes classifiers
TABLE I. SUMMARY OF THE OPTIMAL COLOR SPACES FOR GIVEN CLASSES, GIVEN DIFFERENT CRITERIA, FOR A TEST SET OF IMAGES FROM THE FESB MLID DATASET Class Precisio Recall [F.sub.1]- [F.sub.2]- [F.sub.0.5]- n measure measure measure 1 HSV CIE-XYZ CIE-XYZ CIE-XYZ HSV (14.81%) (26.31%) (15.263%) (20.403%) (14.808%) 2 HLS CIE YCgCr CIE YCgCr (52.59%) L*a*b* (50.379%) L*a*b* (50.916%) (50.96%) (50.074%) 3 YCbCr RGB RGB RGB YCbCr (82.63%) (86.91%) (82.843%) (85.236%) (82.604%) 4 YCgCr HLS HLS HLS YCgCr (54.51%) (65.06%) (54.406%) (60.334%) (54.113%) 5 YCbCr HLS YCbCr HLS YCbCr (11.7%) (29.5%) (16.464%) (22.083%) (13.231%) 6 HSV CIE CIE CIE HSV (31.4%) L*a*b* L*a*b* L*a*b* (31.62%) (37.19%) (32.683%) (35.246%) 7 HLS HLS HLS HLS HLS (30.48%) (30.64%) (30.56%) (30.608%) (30.512%) 8 CIE YCbCr HSV YCbCr HSV L*a*b* (35.72%) (42.682%) (38.032%) (49.143%) (54.99%) 9 HSV CIE-XYZ CIE CIE HLS (61.66%) (63.83%) L*a*b* L*a*b* (61.099%) (61.122%) (61.545%) 10 CIE CIE CIE CIE CIE L*a*b* L*u*v* L*a*b* L*a*b* L*a*b* (45.75%) (56.61%) (50.016%) (52.981%) (47.366%) 11 CIE-XYZ YCgCr CIE CIE CIE (44.83%) (31.30%) L*a*b* L*a*b* L*a*b* (24.441%) (27.828%) (21.79%) TABLE II. AVERAGE PRECISION, RECALL AND [F.sub.1]-MEASURE FOR MEDITERRANEAN LANDSCAPE IMAGE CLASSIFICATION IN DIFFERENT COLOR SPACES Color space Precision Recall [F.sub.1]-measure RGB 31.13 31.92 29.33 CIE-XYZ 32.15 30.67 27.16 Ohta 33.82 34.23 33.44 HSV 39.39 40.94 39.28 hls 39.07 40.53 38.51 YCgCr 37.54 38.42 37.07 YCbCr 38.22 39.30 37.97 CIE L*a*b* 38.92 40.13 38.66 CIE L*u*v* 37.79 39.07 37.78 TABLE III. TABLE SHOWS HOW LIKELY IS IT THAT A PARTICULAR CLASS WILL APPEAR IN A PARTICULAR PART (TOP, MIDDLE, BOTTOM) OF THE IMAGE IN THE TRAIN IMAGES FROM THE FESB MILD DATASET Class Top Middle Bottom Smoke 0.31 0.53 0.16 Clouds and fog 0.69 0.24 0.07 Sun and light effects 0.83 0.11 0.05 Sky 0.86 0.14 0.00 Water surfaces 0.16 0.27 0.57 Distant landscape 0.38 0.51 0.12 Stone 0.34 0.52 0.14 Distant vegetation 0.11 0.42 0.47 Close vegetation 0.05 0.34 0.61 Low vegetation and agricultural areas 0.01 0.38 0.61 Buildings and artificial objects 0.05 0.20 0.74 TABLE IV. AVERAGE PRECISION, RECALL AND [F.sub.1]-MEASURE FOR MEDITERRANEAN LANDSCAPE IMAGE CLASSIFICATION IN DIFFERENT COLOR SPACES Step [F.sub.1]-measure Percentage of unclassified pixels Step 3 25.05% 74% Step 4 44.80% 22% Step 5 43.78% 10% Step 6 44.14% 0% TABLE V. COMPARISON OF THE PROPOSED CCB EXPERT SYSTEM WITH DIFFERENT CLASSIFIERS. TESTING TIME IS GIVEN IN HOURS, MINUTES AND SECONDS Classifier [F.sub.1]-measure Execution time Normal Bayes 41.25% 00:15:14 MLP 2.66% 00:05:30 CCB expert system 44.14% 00:03:38
|Printer friendly Cite/link Email Feedback|
|Author:||Braovic, Maja; Stipanicev, Darko; Krstinic, Damir|
|Publication:||Advances in Electrical and Computer Engineering|
|Date:||May 1, 2017|
|Previous Article:||Speech rate control for improving elderly speech recognition of smart devices.|
|Next Article:||Three-dimensional spatial-spectral filtering based feature extraction for hyperspectral image classification.|