Printer Friendly

Multiple Attribute Decision Making Based on Cross-Evaluation with Uncertain Decision Parameters.

1. Introduction

Multiple attribute decision making (MADM) is an important part of modern decision science [1-4]. It assumes that there exists a set of alternatives with multiple attributes which a decision maker (DM) should evaluate and analyze. The aim of MADM is to find the most desirable alternative or rank the feasible alternatives for supporting decision makings. As an active research area, MADM problems have been tried to be solved by some classical methods such as the simple additive weighting method (SAW), the analytic hierarchy process (AHP), and the technique for order preference by similarity to ideal solution (TOPSIS) [5-7]. It also has been extensively applied to many aspects such as economics, management, engineering, and technology [8-15].

In MADM problems, the needed decision making information includes attribute values and attribute weights. The attribute values describe characteristics, qualities, and performances of alternatives. The attribute weights are used to measure the importance of attributes. Due to the complexity of the real world and limited knowledge and perception capability of human beings, both attribute values and attribute weights are uncertain [16, 17]. The decision makers are not able to express their preferences or assessments explicitly [18]. In order to deal with such problems, many researchers try to describe attribute values and attribute weights accurately in uncertain environments. Generally, the researches in this area can be classified into three categories as follows.

Firstly, fuzzy set theory that appeared in 1965 has gradually become the mainstream in the field of representing and handling uncertain attribute values [19]. Since the inception of the fuzzy set theory, various extensions have been proposed, such as interval-valued fuzzy set [20], fuzzy multiset [21], intuitionistic fuzzy set (IFS) [22], linguistic fuzzy set [23], and hesitant fuzzy set (HFS) [24].

Secondly, several approaches have been proposed to determine attribute weights. Generally, they fall into three categories: subjective, objective, and hybrid methods [25]. The subjective methods utilize the preferences of a decision maker to select attribute weights [26-28]. The objective methods determine attribute weights based on the objective information [29, 30]. Hybrid methods [31-34] combine both the preferences of a decision maker and objective information to determine attribute weights.

Thirdly, as a nonparametric programming efficiency-rating technique for evaluating a set of decision making units (DMUs) with multiple inputs and outputs, date envelopment analysis (DEA) requires relatively less additional information [35]. DEA has been considered as an effective tool to solve MADM problems without information on weights. For instance, Doyle [36] presents an approach based on DEA method, which derives a set of attribute weights from the data itself. Bernroider and Stix [37] introduce an approach based on DEA with restricted multipliers for accountable and understandable MADM. Wu and Liang [38] develop a new multiple attribute ranking method based on DEA game cross-efficiency model, in which each alternative is viewed as a player that seeks to maximize its own performance.

Although existing methods handle uncertainty to some extent, some drawbacks are also exposed which are listed as follows. First, both different defuzzy approaches and weight determination methods may generate different attribute weights and produce different optimal solutions [39, 40]. Therefore, it is difficult to identify which optimal solution is the best one. Second, behavior factors are not considered in existing MADM methods which may lead to deviation results in decision making. Specially, they do not avoid the bad situation that the ranking results may be determined by a single attribute with an exorbitant weight. In other words, the alternative rank order is consistent with the rank order of overestimated attribute which means that other attributes are totally useless in evaluation. Obviously, such result is unfair and unreasonable because the optimal alternative should be determined by multiple attributes instead of an individual attribute. Third, although DEA models can evaluate MADM problems without weight information, it also suffers from the following three drawbacks. One is that the optimal weights derived by DEA may not be consistent with the opinions of the decision maker [41]. Another shortage is that DEA identifies too many efficient alternatives and cannot well distinguish them [42, 43]. Furthermore, the nonuniqueness solution phenomenon is also common in DEA based MADM methods [44].

Considering the above analysis, we attempt to develop a cross-evaluation based approach to solve MADM problems with uncertain decision information. To be specific, we firstly define and analyze the attribute weight space which contains all feasible weights. Specially to avoid the single dominating attribute situation, we calculate the upper bound weight of each attribute. Then, considering a condition where the attribute values are certain and the attribute weights are uncertain, the return and risk of closeness indexes are calculated based on cross-evaluation in which each alternative seeks to maximize its closeness index by selecting a set of optimal weights. And the coefficient of variation is utilized to integrate the results and produce an alternative rank order. In a more generalized condition where the attribute values are denoted by interval values or ordinal values, the cross-evaluation models considering DM's risk preference are developed.

The main contributions of the current paper can be summarized as follows: (1) we firstly defined the attribute weight space and determined the upper bound weight of each attribute to conduct the uncertain information; (2) both mean and deviation measures of closeness index under cross-evaluation are considered to integrate a unique solution to the MADM problem; (3) the proposed method is extended to more generalized situations where the attribute values are described by interval values and ordinal values, respectively.

The remainder of this paper is organized as follows. In Section 2, we briefly review the relative works. In Section 3, we propose a cross-evaluation based method to solve MADM problems with uncertain weight information. In Section 4, the proposed method is extended to more generalized forms where the attribute information is expressed by interval values and ordinal values. Section 5 illustrates the methods with reported MADM studies and a case study of the selection of community service companies. Conclusions and future research will be displayed in Section 6.

2. Preliminaries

2.1. Formulation of MADM Problems. Suppose X = {[X.sub.2], ..., [X.sub.m]} is a discrete set of alternatives and Y = {[Y.sub.1], [Y.sub.2], ..., [Y.sub.n]} is a set of n attributes with the weight vector w = ([w.sub.1], [w.sub.2], ..., [w.sub.n])T, where [w.sub.j] > 0, [[summation].sup.n.sub.j=1] [w.sub.j] = 1. Let A = [([a.sub.ij]).sub.mxn] be a decision matrix, where [a.sub.ij] denotes the attribute value that the ith alternative [X.sub.i] [member of] X takes with respect to the jth attribute [Y.sub.j] [member of] Y.

In general, there are both benefit type attributes (i.e., the larger the better) and cost type attributes (i.e., the smaller the better) in a MADA problem. We transform the original decision matrix A = [([a.sub.ij]).sub.mxn] into a normalized matrix R = [([r.sub.ij]).sub.mxn] by using the method from Hwang and Yoon [7]:

[mathematical expression not reproducible]. (1)

2.2. Characteristics of the TOPSIS Method. The basic principle of TOPSIS method is that the optimal alternative should have both the shortest distance from the positive ideal solution and the furthest distance from the negative ideal solution [14].

The TOPSIS procedure is performed in six stages as follows:

(a) Normalize the decision matrix

[r.sub.ij] = [a.sub.ij]/[square root of ([[summation].sup.n.sub.j=1])], i=1, 2, ..., m; j = 1, 2, ..., n, (2)

where [a.sub.ij] is the jth attribute value of alternative i.

(b) Calculate the weighted normalized decision matrix

[v.sub.ij] = [w.sub.j][r.sub.ij], i= 1, 2, ..., m; j = 1, 2, ..., n, (3)

where [w.sub.j] is the weight of the jth attribute, and [[summation].sup.n.sub.j=1] [w.sub.j] = 1.

(c) Determine the positive ideal and negative ideal solution

Positive ideal solution [A.sup.+] = {[v.sup.+.sub.1], [v.sup.+.sub.2], ..., [v.sup.+.sub.n]}

[mathematical expression not reproducible],

Negative ideal solution [A.sup.-] = {[v.sup.-.sub.1], [v.sup.-.sub.2], ..., [v.sup.-.sub.n]}

[mathematical expression not reproducible], (4)

where I' indicate the index set of the benefit type attributes and I" the index set of the cost type attributes.

(d) Measure the distances from the positive ideal solution and negative ideal solution, respectively:

[mathematical expression not reproducible] (5)

(e) Calculate the relative closeness to the ideal solutions

[C.sup.*.sub.i] = [D.sup.-.sub.i]/([D.sup.+.sub.i] + [D.sup.-.sub.i]), i = 1, 2, ..., m. (6)

(f) Rank the alternatives.

The larger the closeness index of an alternative, the higher the rank of the alternative.

2.3. DEA Cross-Efficiency Evaluation. We assume that there are n DMUs to be evaluated in terms of m inputs and n outputs. The ith input and rth output of [DMU.sub.j] are denoted by [X.sub.ij] (i = 1, ..., m) and [y.sub.rj], respectively. The efficiency of a [DMU.sub.d] can be calculated by the following CCR model in linear programming formulation [35]:

[mathematical expression not reproducible]. (7)

When the above model is solved, a set of optimal weights [w.sup.*.sub.1d], ..., [w.sup.*], [u.sup.*.sub.1d], ..., [u.sup.*] is obtained. Then the d cross-efficiency of any [DMU.sub.j] is computed as [38]

[mathematical expression not reproducible]. (8)

For [DMU.sub.j] (j = 1, ..., n), the average of all [] (d = 1, ..., n), denoted by [[bar.E].sub.j] = (1/n) [[summation].sup.n.sub.d=1] [], is considered as the cross-efficiency score of [DMU.sub.j].

3. Proposed Method

In this section, we will introduce the so-called single dominating attribute issue and propose a model to support decisions by incorporating considerations on this issue in Section 3.1; furthermore, in Section 3.2, we will introduce the cross-evaluation framework and the accompanying risk consideration; finally, in Section 3.3, we propose a procedure summarizing all the aforementioned considerations for MADM.

3.1. Determination of the Weight Space. We observe that, in MADM problems, an individual attribute can determine the final ranks of all alternatives when the corresponding weight is greater than some threshold value. In other words, the rank order of the alternatives is completely in accordance with the rank order of the dominating attribute and the rest of the attributes are totally ignored in the evaluation. It should be noted that the ignored attributes are taken into the evaluation only because the decision maker regards them as import factors or they could have excluded them from the decision process in the first place. Therefore, we argue this is unacceptable. In order to address this issue, it is necessary to define and compute the threshold upper bound of each attribute weight. The upper bound can aid the decision process by suggesting meaningful weights.

For the simplicity of exposition, if the attribute values [r.sub.ik] (i = 1, ..., m) in the kth column of the decision matrix R have the relation [mathematical expression not reproducible], then we denote the ranks of the alternatives according to the values of the kth attribute [Y.sub.k] as [mathematical expression not reproducible]. With this notation in position, the single dominating attribute happens if the final rank order of all alternates is the same as some [mathematical expression not reproducible] no matter how the decision maker distributes the relative weights of the other attributes. We formulize this notion as Definition 1 as below.

Definition 1. Let k be the index of some attribute. [w.sup.*.sub.k] is the threshold upper bound of [w.sub.k] if and only if the following conditions are satisfied:

(a) If [w.sub.k] [greater than or equal to] [w.sup.*.sub.k], then [for all]1 [less than or equal to] i, t [less than or equal to] m, [r.sub.ik] [greater than or equal to] [] if and only if [[summation].sup.n.sub.j=1] [w.sub.j][r.sub.ij] [greater than or equal to] [[summation].sup.n.sub.j=1] [w.sub.j] [].

(b) If [w.sub.k] < [w.sup.*.sub.k], then [there exists]1 [less than or equal to] i, t [less than or equal to] m, such that [r.sub.ik] [greater than or equal to] [] and [[summation].sup.n.sub.j=1] [w.sub.j] [r.sub.ij] [less than or equal to] [[summation].sup.n.sub.j=1] [w.sub.j][] hold true simultaneously.

To implement Definition 1, we provide Theorem 2 below to determine the threshold upper bound.

Theorem 2. The threshold upper bound of the kth attribute ([w.sup.*.sub.k]) is given by the minimization program

[mathematical expression not reproducible] (9)

[mathematical expression not reproducible] (10)

where [w.sup.(f).sub.f] = 1 - [w.sub.k] if j is the fth element of the set {1, ..., N] \ [k] in ascending order; otherwise, [w.sup.(f).sub.j] = 0. Let us take k = 1 as an example. We have ([w.sup.(1).sub.2], ([w.sup.(1).sub.3], ..., ([w.sup.(1).sub.n]) = (1 - [w.sub.1], 0, ..., 0), (([w.sup.(2).sub.2], ([w.sup.(2).sub.3], ..., ([w.sup.(2).sub.n]) = (0, 1 - [w.sub.1], 0, ..., 0), ..., ([w.sup.(n-1).sub.2]), ([w.sup.(n-1).sub.3]), ..., ([w.sup.(n-1).sub.n])) = (0, ..., 0, 1 - [w.sub.1]). Thesub script [i.sub.h] (h [member of] [1, ..., m]) of r in the above model indicates the [i.sub.h] preferred alternative determined by the preference relation [mathematical expression not reproducible].

Proof. Clearly, there are (n - 1) constraints in (9). When k = 1, the weight combinations ([w.sup.(2).sub.2], [w.sup.(2).sub.3], ..., [w.sup.(2).sub.n]) = (1 - [w.sub.1], 0, ..., 0), ([w.sup.(2).sub.2], [w.sup.(2).sub.3], ..., [w.sup.(2).sub.n]) = (0, 1 - [w.sub.1], 0, ..., 0), ..., ([w.sup.(n-1).sub.2], [w.sup.(n-1).sub.3], ..., [w.sup.(n-1).sub.n]) = (0, ..., 0, 1 - [w.sub.1]) are applied to the n-1 constraints in order. The first constraint guarantees that the alternatives rank result has nothing to do with attribute [Y.sub.2]. Similarly, the remaining n - 2 constraints ensure that the alternatives rank result has nothing to do with the other attributes [Y.sub.3], ..., [Y.sub.n]. So the optimal minimization [w.sub.1] satisfies (a) in Definition 1.

Conversely, we suggest that a constraint belonging to the [] (I = 1, ..., n - 1) group is removed from the model. Then, attribute Yi may play a part in the ranking because it is not completely suppressed. In other words, (b) in Definition 1 holds.

Similarly, the upper bound of each attribute weight ([w.sup.*.sub.k], k = 2, ..., n) can be calculated by the model in Theorem 2 with k = 2, ..., n. This proof is completed.

Example 3. Ma et al. [31] consider a case of a robot user expecting to select a robot and there are four alternatives for him to choose. There are four attributes to evaluate them, that is, cost, velocity, repeatability, and load capacity. The normalized decision matrix is as follows:

[mathematical expression not reproducible]. (11)

Suppose that the information on attribute weights is completely unknown. The upper bound weight of the first attribute cost can be calculated by the above model.

By calculating the above model provided in Theorem 2, we obtain the optimal value [w.sup.*.sub.1] = 2/3 denoting the threshold upper bound of the first attribute [w.sub.1]. Analogously, each upper bound of attribute weight can be calculated by similar models, and the values are [w.sup.*.sub.2] = 10/13, [w.sup.*.sub.3] = 4/5, [w.sup.*.sub.4] = 9/14.

3.2. Mean and Deviation of Closeness Index Based Cross-Evaluation Models. In this subsection, in the environment of uncertain attribute weights, we develop cross-evaluation models to calculate the mean and deviation of closeness index for each assessed alternative, respectively. Then, the computed indexes are integrated to obtain the evaluation results.

Cross-efficiency evaluation method, since being proposed by Sexton et al. [45], has been used to evaluate and rank DMUs. Its main idea is to use cross-evaluation instead of a self-evaluation. There are at least two advantages for cross-evaluation method. One is that it obtains a unique ranking among the DMUs. The other is that the cross-efficiency means can act effectively to differentiate between good and bad alternatives [46]. Hence, the cross-evaluation method is widely used for ranking performance of alternatives, for example, equipment design [47], supplier selection [48], and stock market [49].

To calculate the self-evaluation closeness index for each alternative, we consider the following mathematical programming problem for a given alternative [X.sub.i]:

max [D.sup.-.sub.i]/[D.sup.-.sub.i] + [D.sup.+.sub.i] = [C.sup.max.sub.i]

w [member of] [OMEGA] (w). (12)

Here, [D.sup.-.sub.i] and [D.sup.+.sub.i] represent the distance from negative ideal point and ideal point (defined in (5)), respectively. For ease of calculation, the ideal point and negative point are defined as [A.sup.+] = [[w.sub.1], [w.sub.2], ..., [w.sub.n]] and [A.sup.-] = [0, 0, ..., 0]. Then [D.sup.+.sub.i] and [D.sup.-.sub.i] are denoted by [D.sup.+.sub.i] = [square root of ([[summation].sup.n.sub.j=1] [([w.sub.j] [r.sub.ij] - [w.sub.j]).sup.2] and [D.sup.-.sub.i] = [square root of ([[summation].sup.n.sub.j=1] [([w.sub.j][r.sub.ij]).sup.2], respectively. The constraint w [member of] [OMEGA](w) depicts the possible set of attribute weights. Obviously, when the weights information of attributes is completely unknown, [OMEGA](w) = [[w.sub.j] > 0, [[summation].sup.n.sub.j=1] [w.sub.j] = 1]. Introducing the upper bound weight information calculated in Section 3.1, [OMEGA](w) = [[w.sub.j] < [w.sup.*.sub.j], [[summation].sup.n.sub.j=1] [w.sub.j] = 1].

In many practical situations, although the weight values are difficult to obtain, the weight information is partially known and predefined by the weight space. As follows, we introduce two types of partial weight information as references to decision maker.

Type 1 (pairwise comparison weights information). In MADM, attribute weights [w.sub.j] are mainly introduced to reflect the importance to attributes [Y.sub.j] (j = 1, 2, ..., n). As stated in Section 1, pairwise comparison is a widely accepted subjective method to determine attribute weight.

Attribute [Y.sub.2] is at least two times and not more than three times as important as attribute [Y.sub.1].

In notational jargons, this is denoted by

2[w.sub.1] [less than or equal to] [w.sub.2] [less than or equal to] 3[w.sub.1]. (13)

In general terms, we compare [Y.sub.1] with [Y.sub.j] (j = 2, 3, ..., n), respectively. The bound linear constraints are denoted by

[l.sub.j] [w.sub.1] [less than or equal to] [w.sub.j] [less than or equal to] [u.sub.j][w.sub.1], (14)

in which [l.sub.j] and [u.sub.j] represent the lower and upper bounds of times [w.sub.j] compared with [w.sub.1], respectively. Hence, the attribute weights space is obtained by the above pairwise comparison information, which is expressed as [OMEGA](w) = {[w.sub.j] > 0, [l.sub.j] [w.sub.1] [less than or equal to] [w.sub.j] [less than or equal to] [u.sub.j] [w.sub.1], j = 1, 2, ..., n}. Clearly, [l.sub.1] = [u.sub.1] = 1 holds.

Compared with conventional subjective weight determination approaches such as the AHP, weights space is more feasible and applicative because the decision makers need not express their preferences explicitly. In many situations, experts are pleased to give such interval information rather than exact value due to the complexity of actual problems.

Type 2 (ordinal weights information). Outranking relation [50], usually denoted by S, was proposed by Roy whose aim was to establish a realistic representation of the three basic preference situations: strict preference, indifference, and incomparability. Motivated by the widely accepted outranking relation, we introduce the concept of rank-order weights information. In many situations, although the weight values are difficult to give by decision makers, the ranking of each attribute is easy to obtain according to importance. Take the Olympic Games as an example, the achievements of athletes and nations are judged by gold, silver, and bronze medals. Apparently, a gold medal is more important than a silver medal, and a silver medal is more important than a bronze medal. Then, the weights space can be denoted by [OMEGA](w) = {[w.sub.1] > [w.sub.2] > [w.sub.3], [w.sub.1], [w.sub.2], [w.sub.3] > 0}, in which [w.sub.1], [w.sub.2], [w.sub.3] represent the weight of gold, silver, and bronze medal, respectively.

The above self-evaluation model (see (12)) shows that all alternatives choose their own attribute weights in order to maximize their own self-evaluation closeness index. Similar to the form of DEA cross-efficiency, we obtain a set of optimal attribute weights [W.sup.*.sub.i] = ([w.sup.*.sub.i1], ..., [w.sup.*]) according to the above model. When these optimal weights are applied to evaluate alternative [X.sub.k], a cross score denoted by [C.sub.ik] = [D.sup.-.sub.ik]/([D.sup.-.sub.ik] + [D.sup.+.sub.ik]) is obtained.

For alternative k (k = 1, ..., m), we use the average of all [C.sub.ik] to measure the expected return of cross-evaluation closeness index, which is written as

r([X.sub.k]) = 1/m [m.summation over (i=1)] [C.sub.ik]. (15)

Furthermore, we use the standard variance

[delta] ([X.sub.k]) = [square root of ([[summation].sup.m.sub.i=1][([C.sub.ik] - r([X.sub.k])).sup.2]/n)] (16)

to measure the risk caused by different sets of optimal weights produced by each alternative.

In general, there are both return of closeness index (defined by the mean of cross-evaluation score) and risk of closeness index (defined by the deviation between return and each cross-evaluation closeness index) in cross-evaluation framework. It is not hard to see that the larger return score the better and the smaller risk score the better. Specifically, given two alternatives [X.sub.s] and [X.sub.t], if R([X.sub.s]) > R([X.sub.t]) and [delta]([X.sub.s]) < [delta]([X.sub.t]), then Xs > [X.sub.t] ([X.sub.s] is better than [X.sub.t]) clearly holds. However, when R([X.sub.s]) > R([X.sub.t]) and [delta]([X.sub.s]) > [delta]([X.sub.t]), it is not easy to compare the two alternatives directly.

Then, we integrate the return and risk measures by the well-known coefficient of variation [51]

C. V ([X.sub.k]) = [delta]([X.sub.k])/r([X.sub.k]) x 100%. (17)

For two given alternatives, one having a smaller coefficient of variation is considered to be better than the other one. Thus, all the alternatives are ranked by the comparison of their corresponding coefficient of variation.

3.3. Procedure of the Proposed Approach. Based on the above analysis, the procedure for generating a solution to the MADM problem is elaborated in the following.

Step 1 (formulation of a MADM problem). The decision maker identifies n attributes to evaluate m alternatives. The original attribute values are standardized by (1).

Step 2 (obtainment of the attribute weight space). First, the upper bounds of each attribute are calculated by Theorem 2 to guarantee the effectiveness of all the attributes. Then, different types of constraint of attribute weights are determined by the decision maker to reflect the importance of each attribute. Last, the weight space including all the feasible weight combinations is obtained.

Step 3 (calculation of the mean and deviation measures). Firstly, the maximized closeness index of each alternative is calculated by (12). Secondly, the mean measure of each alternative's cross closeness index is computed by (15). Third, the deviation measure of each alternative's cross closeness index is obtained by (16).

Step 4 (evaluation of the alternatives via coefficient of variation). Since the mean and deviation measures of each alternative's cross closeness index are obtained in Step 3, the coefficient of variation defined in (17) is utilized to integrate the results.

Step 5 (generation of rank order of the alternatives). Through comparing the coefficients of variation among the alternatives, the rank order can be directly obtained.

4. Extension with Uncertain Attribute Values

4.1. Model with Interval Attribute Value. It is widely acknowledged that uncertainty is associated with attribute values due to complexity of real world and limited knowledge of human beings. In many situations, the attribute value [r.sub.ij] is uncertain, but we know that the feasible attribute value [r.sub.ij] [member of] [[r.sup.-.sub.ij], [r.sup.+.sub.ij]]. Here, we present the following self-evaluation model with interval attribute values [52] and uncertain attribute weights:

[mathematical expression not reproducible], (18)

in which [mathematical expression not reproducible] represent distance of upper bounds from ideal point, distance of upper bounds from negative ideal point, distance of lower bounds from ideal point, and distance of lower bounds from negative ideal point, respectively. [alpha] and [beta] reflect the risk coefficients that combine upper closeness index and lower closeness, respectively. In practice, [alpha] and [beta] are decided by the decision maker according to his or her risk preference. Specifically, the decision maker is neutral if [alpha] = [beta] = 1/2; [alpha] > [beta] and [alpha] < [beta] indicate that he is optimistic and pessimistic, respectively.

4.2. Model with Ordinal Attribute Value. Another form of uncertain preferential information is expressed by ordinal attribute value (e.g., "more preferable" and "less preferable"). It is argued to be more stable and more reliable than cardinal value in some situations [53]. In this paper, an approach of threshold value is introduced to transform ordinal value into linear formulation. For instance, given four attributes a, b, c, and d, the ordinal values are 1, 2, 3, and 4, respectively. By specifying a threshold value 0.1, the attribute values can be converted to a cardinal attribute space [OMEGA](a, b, c, d) = {a = 1, a - b [greater than or equal to] 0.1, b - c [greater than or equal to] 0.1, c - d [greater than or equal to] 0.1, d = 0}. In general, let the ordinal attribute values [a.sub.ij] (i = 1, ..., m) have relation [mathematical expression not reproducible], and then the cardinal attribute space [mathematical expression not reproducible] is obtained ([epsilon] is the threshold value given by the decision maker). Here, we present the following self-evaluation model with ordinal attribute values and uncertain attribute weights:

[mathematical expression not reproducible] (19)

[mathematical expression not reproducible]; (20)

[mathematical expression not reproducible]; (21)

For each alternative [X.sub.k](k = 1, ..., m) under evaluation, we obtain a group of optimal attribute values and a set of attribute weights. They are denoted by [A.sup.*.sub.k] = [([r.sup.k.sub.ij]).sub.mxn] and [W.sup.*.sub.k] = ([w.sup.k.sub.1], [w.sup.k.sub.2], ..., [w.sup.k.sub.n]), respectively. When the optimal decision information is applied to evaluate alternative [X.sub.l] (l = 1, ..., m), a cross score denoted by [mathematical expression not reproducible] is obtained.

Similar to that defined by (14) and (15), the mean and deviation measures of alternative [X.sub.l] are denoted by r([X.sub.l]) = [[summation].sup.m.sub.k=1] [c.sub.kl] and [delta]([X.sub.l]) = [square root of ([[summation].sup.m.sub.k=1] [([C.sub.kl] - r([X.sub.l])).sup.2]/n)], respectively.

Note that (12), (18), and (19) are nonlinear models; they can be transformed into linear forms according to practical cases.

5. Illustrative Examples

In order to illustrate the applications of our proposed methods, three practical MADM problems are presented.

5.1. Application to House Selection with Cardinal Attribute Values. Xu [54] stated a purchase house problem: there are four possible houses to be selected, and the attributes are as follows: (1) [Y.sub.1] is the house price (ten thousand yuan); (2) [Y.sub.2] is the usable area (square meter); (3) [Y.sub.3] is the distance between house and the center of city (kilometers); (4) [Y.sub.4] is the nature environment (scores), where [Y.sub.2] and [Y.sub.4] are benefit attributes and [Y.sub.1] and [Y.sub.3] are cost attributes. The attribute weights are completely unknown. The decision making matrix A and normalized decision making matrix R are listed in Tables 1 and 2, respectively. Step 1 is completed.

By utilizing (9) and (10), the upper bound of each attribute weight is obtained. And as a result, the attribute weight space is [OMEGA](w) = {0 < [w.sub.1] < 0.77, 0 < [w.sub.2] < 0.78, 0 < [w.sub.3] < 0.62, 0 < [w.sub.4] < 0.62, [w.sub.1] + [w.sub.2] + [w.sub.3] + [w.sub.4] = 1}. Step 2 is completed.

In terms of Step 3 in Section 3.3, we obtain the cross-evaluation measurements of each alternative, which are presented in Table 3. Step 3 is completed.

Based on Table 3 and the calculation of coefficient of variation, the results are shown in Table 4.

Therefore, we can rank the houses in accordance with the CV values presented in Table 4 to get a complete rank order: [X.sub.4] > [X.sub.2] > [X.sub.1] > [X.sub.3]. And [X.sub.4] is the best house under evaluation. Step 4 is completed.

In order to understand our approach more clearly, we compare our approach with other three similar methods [54], namely, average attribute value method, maximizing deviation method, and information entropy method, respectively, with the same example. The corresponding results of such three methods are listed in Table 5.

According to Tables 4 and 5, the result derived from our cross-evaluation approach is quite different from that of the existing three weight determination methods. In fact, Table 5 shows that the three weight determination methods generate three attribute weight vectors and produce three different optimal solutions. However, there is no single method that can guarantee more convincing weight vector than others. Hence, we treat the attribute weight as a weight space rather than a certain value in this paper. The comparison result validates that our proposed approach can overcome the shortage that we cannot obtain the most dominant result caused by the diversity of the way of weight vector determined.

5.2. Application to University Faculty with Interval Attribute Value. Consider the example in Bryson and Mobolurin's [55] study; that is, five faculty candidates (A, B, C, D, E) are evaluated for tenure and promotion through three attributes: teaching, research, and service. According to the DM's preference, the information for both attribute and alternatives is given by interval numbers, as shown in Tables 6 and 7, respectively (for the purpose of illustration, our data is different from the literature).

Table 8 shows the computed results of the proposed approach with interval values. The self-evaluation score of each alternative is listed in the second column and the cross-evaluation scores are listed in columns 3-7. Columns 8 and 9 show the mean and deviation measures, respectively. Then, the results in column 10 are applied to generate a rank order of the five candidates A > C > E > B > D, which is a solution to the problem.

5.3. Application to the Selection of Community Service Companies with Ordinal Attribute Values. In this subsection, we apply our approach to the evaluation and selection of five community service companies that are located in Hefei, a city in Anhui province of China. These companies provide various kinds of service for the public, such as property management, catering, health care, and entertainment. In the case study, the decision maker is an official from the development and reform commission of Hefei, with the help of a group of experts from the governments, relevant enterprises, and community organizations. Four attributes are used to evaluate the six companies. They are quality of service ([Y.sub.1]), diversity of service ([Y.sub.2]), profitability ([Y.sub.3]), and development potential (Y4). The original attribute values are given by ordinal values, as shown in Table 9. The decision procedures are as follows.

After considering the four attributes, the decision maker views that quality of service ([Y.sub.1]) is the most important attribute in evaluation. The development potential ([Y.sub.4]) is the second most important one, profitability ([Y.sub.3]) is the third most important one, and diversity of service ([Y.sub.2]) is the least important one, respectively. And the difference between two adjacent attribute weights should be more than 0.1. In addition, each weight value should be no more than 0.5. Then the weight space is denoted by [mathematical expression not reproducible].

By specifying a threshold value of 0.1, ordinal attribute values are transformed into four sets of attribute value spaces.

They are denoted by

[mathematical expression not reproducible]. (22)

Table 10 shows the computed results of the proposed approach. The second column shows the self-evaluation score of each alternative computed by (19). And columns 3-7 list the cross-evaluation scores. Then the mean and deviation measures of each alternative are presented in columns 8 and 9, respectively. Consequently, the results in column 10 are applied to generate a rank order of the five community service companies [X.sub.5] > [X.sub.1] > [X.sub.2] > [X.sub.4] > [X.sub.3]. The decision maker is pleased to select company X5 with the best performance among the five alternatives.

6. Conclusions and Future Research

Due to the complexity and uncertainty of MADM problems in real world, the decision maker is not able to provide the exact attribute weights. And different weight determination methods are difficult to generate a most satisfaction solution. The current paper presents a new MADM method based on cross-evaluation, in which the weight information is expressed by the so-called weight space. It attempts to rectify some shortcomings of existing MADM methods by providing the following important features: (1) it is more feasible to provide a weight space rather than specifying a set of weight values; (2) the single dominating attribute drawback can be avoided by the restriction of the upper bound of each attribute; (3) compared with DEA based methods, the variable attribute weights are consistent with the opinion of decision maker; (4) both mean score and the deviation score are considered in our cross-evaluation models; and (5) the proposed method is also successfully extended to more generalized situations in which attribute values are expressed by interval values and ordinal values, respectively.

We have illustrated the applicability of the proposed methods by revisiting two reported MADM studies and by depicting a case study on the selection of community service companies. Although our proposed method makes some improvement in solving MADM problems, it only can deal with the individual decision making case. Future research may attempt to employ our cross-evaluation methods to handle group decision making problem.

Competing Interests

The authors declare that they have no competing interests.


The research is supported by National Key Technology Support Program (no. 2014BAH27F00), National Natural Science Funds of China (no. 71471053), and the Fundamental Research Fund for the Central Universities (no. JZ2014HGBZ0339).


[1] R. H. Green and J. R. Doyle, "On maximizing discrimination in multiple criteria decision making," Journal of the Operational Research Society, vol. 46, no. 2, pp. 192-204, 1995.

[2] R. E. Steuer and P. Na, "Multiple criteria decision making combined with finance: a categorized bibliographic study," European Journal of Operational Research, vol. 150, no. 3, pp. 496-515, 2003.

[3] J. Wallenius, J. S. Dyer, P. C. Fishburn, R. E. Steuer, S. Zionts, and K. Deb, "Multiple criteria decision making, multi-attribute utility theory: recent accomplishments and what lies ahead," Management Science, vol. 54, no. 7, pp. 1336-1349, 2008.

[4] W. Ho, X. Xu, and P. K. Dey, "Multi-criteria decision making approaches for supplier evaluation and selection: a literature review," European Journal of Operational Research, vol. 202, no. 1, pp. 16-24, 2010.

[5] C. W. Churchman, R. L. Ackoff, and N. M. Smith, "An approximate measure of value," Journal of the Operational Research Society of America, vol. 2, no. 2, pp. 172-187, 1954.

[6] T. L. Saaty, "The analytic hierarchy process," in Proceedings of the Second International Seminar on Operational Research in the Basque Provinces, vol. 4, pp. 189-234, 1980.

[7] C. L. Hwang and K. Yoon, Multiple Attribute Decision Making: Methods and Applications, Springer, Berlin, Germany, 1981.

[8] H. Deng, C.-H. Yeh, and R. J. Willis, "Inter-company comparison using modified TOPSIS with objective weights," Computers and Operations Research, vol. 27, no. 10, pp. 963-973, 2000.

[9] P. K. Dey, "Integrated project evaluation and selection using multiple-attribute decision-making technique," International Journal of Production Economics, vol. 103, no. 1, pp. 90-103, 2006.

[10] G.-D. Li, D. Yamaguchi, and M. Nagai, "A grey-based decision-making approach to the supplier selection problem," Mathematical & Computer Modelling, vol. 46, no. 3-4, pp. 573-581, 2007.

[11] A. Shanian and O. Savadogo, "TOPSIS multiple-criteria decision support analysis for material selection of metallic bipolar plates for polymer electrolyte fuel cell," Journal of Power Sources, vol. 159, no. 2, pp. 1095-1104, 2006.

[12] G.-H. Tzeng, Y.-P. O. Yang, C.-T. Lin, and C.-B. Chen, "Hierarchical MADM with fuzzy integral for evaluating enterprise intranet web sites," Information Sciences, vol. 169, no. 3-4, pp. 409-426, 2005.

[13] T. Yang and C.-C. Hung, "Multiple-attribute decision making methods for plant layout design problem," Robotics and Computer-Integrated Manufacturing, vol. 23, no. 1, pp. 126-137, 2007.

[14] A. Kelemenis and D. Askounis, "A new TOPSIS-based multi-criteria approach to personnel selection," Expert Systems with Applications, vol. 37, no. 7, pp. 4999-5008, 2010.

[15] B. Bairagi, B. Dey, B. Sarkar, and S. K. Sanyal, "A De Novo multi-approaches multi-criteria decision making technique with an application in performance evaluation of material handling device," Computers & Industrial Engineering, vol. 87, pp. 267282, 2015.

[16] J. S. H. Kornbluth, "Dynamic multi-criteria decision making," Journal of Multi-Criteria Decision Analysis, vol. 1, no. 2, pp. 81-92, 1992.

[17] Y.-H. Lin, P.-C. Lee, T.-P. Chang, and H.-I. Ting, "Multi-attribute group decision making model under the condition of uncertain information," Automation in Construction, vol. 17, no. 6, pp. 792-797, 2008.

[18] R. Lahdelma and P. Salminen, "SMAA-2: stochastic multi-criteria acceptability analysis for group decision making," Operations Research, vol. 49, no. 3, pp. 444-454, 2001.

[19] L. A. Zadeh, "Fuzzy sets," Information and Computation, vol. 8, pp. 338-353, 1965.

[20] I. B. Turksen, "Interval valued fuzzy sets based on normal forms," Fuzzy Sets and Systems, vol. 20, no. 2, pp. 191-210, 1986.

[21] S. Miyamoto, "Multisets and fuzzy multisets," in Soft Computing and Human-Centered Machines, Z. Q. Liu and S. Miyamoto, Eds., pp. 9-33, Springer, Berlin, Germany, 2000.

[22] K. T. Atanassov, "Intuitionistic fuzzy sets," Fuzzy Sets and Systems, vol. 20, no. 1, pp. 87-96, 1986.

[23] Z. S. Xu, "Uncertain linguistic aggregation operators based approach to multiple attribute group decision making under uncertain linguistic environment," Information Sciences, vol. 168, no. 1-4, pp. 171-184, 2004.

[24] V. Torra, "Hesitant fuzzy sets," International Journal of Intelligent Systems, vol. 25, no. 6, pp. 529-539, 2010.

[25] C. Kao, "Weight determination for consistently ranking alternatives in multiple criteria decision analysis," Applied Mathematical Modelling. Simulation and Computation for Engineering and Environmental Systems, vol. 34, no. 7, pp. 1779-1787, 2010.

[26] D. Horsky and M. R. Rao, "Estimation of attribute weights from preference comparisons," Management Science, vol. 30, no. 7, pp. 801-822, 1984.

[27] P. A. Bottomley and J. R. Doyle, "A comparison of three weight elicitation methods: good, better, and best," Omega, vol. 29, no. 6, pp. 553-560, 2001.

[28] J. Figueira and B. Roy, "Determining the weights of criteria in the ELECTRE type methods with a revised Simos' procedure," European Journal of Operational Research, vol. 139, no. 2, pp. 317-326, 2002.

[29] L. E. Shirland, R. R. Jesse, R. L. Thompson, and C. L. Iacovou, "Determining attribute weights using mathematical programming," Omega, vol. 31, no. 6, pp. 423-437, 2003.

[30] Y.-M. Wang and Y. Luo, "Integration of correlations with standard deviations for determining attribute weights in multiple attribute decision making," Mathematical & Computer Modelling, vol. 51, no. 1-2, pp. 1-12, 2010.

[31] J. Ma, Z.-P. Fan, and L.-H. Huang, "A subjective and objective integrated approach to determine attribute weights," European Journal of Operational Research, vol. 112, no. 2, pp. 397-404, 1999.

[32] X. Xu, "A note on the subjective and objective integrated approach to determine attribute weights," European Journal of Operational Research, vol. 156, no. 2, pp. 530-532, 2004.

[33] P. J. H. Schoemaker and C. C. Waid, "An experimental comparison of different approaches to determining weights in additive utility models," Management Science, vol. 28, no. 2, pp. 182-196, 1982.

[34] Y.-M. Wang and C. Parkan, "A general multiple attribute decision-making approach for integrating subjective preferences and objective information," Fuzzy Sets and Systems, vol. 157, no. 10, pp. 1333-1345, 2006.

[35] A. Charnes, W. W. Cooper, and E. Rhodes, "Measuring the efficiency of decision making units," European Journal of Operational Research, vol. 2, no. 6, pp. 429-444, 1978.

[36] J. R. Doyle, "Multiattribute choice for the lazy decision maker: let the alternatives decide," Organizational Behavior and Human Decision Processes, vol. 62, no. 1, pp. 87-100, 1995.

[37] E. Bernroider and V. Stix, "A method using weight restrictions in data envelopment analysis for ranking and validity issues in decision making," Computers & Operations Research, vol. 34, no. 9, pp. 2637-2647, 2007.

[38] J. Wu and L. Liang, "A multiple criteria ranking method based on game cross-evaluation approach," Annals of Operations Research, vol. 197, no. 1, pp. 191-200, 2012.

[39] C. Fu and K.-S. Chin, "Robust evidential reasoning approach with unknown attribute weights," Knowledge-Based Systems, vol. 59, no. 2, pp. 9-20, 2014.

[40] B. S. Ahn, "Extreme point-based multi-attribute decision analysis with incomplete information," European Journal of Operational Research, vol. 240, no. 3, pp. 748-755, 2015.

[41] W. W. Cooper, J. L. Ruiz, and I. Sirvent, "Choosing weights from alternative optimal solutions of dual multiplier models in DEA," European Journal of Operational Research, vol. 180, no. 1, pp. 443-458, 2007.

[42] G. R. Jahanshahloo and P. F. Shahmirzadi, "New methods for ranking decision making units based on the dispersion of weights and Norm 1 in Data Envelopment Analysis," Computers & Industrial Engineering, vol. 65, no. 2, pp. 187-193, 2013.

[43] V. E. Krivonozhko, O. B. Utkin, M. M. Safin, and A. V. Lychev, "On some generalization of the DEA models," Journal of the Operational Research Society, vol. 60, no. 11, pp. 1518-1527, 2009.

[44] P. Andersen and N. C. Petersen, "A procedure for ranking efficient units in data envelopment analysis," Management Science, vol. 39, no. 10, pp. 1261-1264, 1993.

[45] T. R. Sexton, R. H. Silkman, and A. J. Hogan, "Data envelopment analysis: critique and extensions," New Directions for Program Evaluation, vol. 1986, no. 32, pp. 73-105, 1986.

[46] A. Boussofiane, R. G. Dyson, and E. Thanassoulis, "Applied data envelopment analysis," European Journal of Operational Research, vol. 52, no. 1, pp. 1-15, 1991.

[47] K. Wang and F. Wei, "Game cross efficiency DEA based project review of equipment design," Journal of Beijing University of Aeronautics and Astronautics, vol. 35, no. 10, pp. 1278-1282, 2009.

[48] T. Wu and J. Blackhurst, "Supplier evaluation and selection: an augmented DEA approach," International Journal of Production Research, vol. 47, no. 16, pp. 4593-4608, 2009.

[49] S. Lim, K. W. Oh, and J. Zhu, "Use of DEA cross-efficiency evaluation in portfolio selection: an application to Korean stock market," European Journal of Operational Research, vol. 236, no. 1, pp. 361-368, 2014.

[50] B. Roy, "The outranking approach and the foundations of electre methods," Theory and Decision, vol. 31, no. 1, pp. 49-73, 1991.

[51] G. F. Reed, F. Lynn, and B. D. Meade, "Use of coefficient of variation in assessing variability of quantitative assays," Clinical & Diagnostic Laboratory Immunology, vol. 9, no. 6, pp. 1235-1239, 2002.

[52] G. R. Jahanshahloo, F. H. Lotfi, and M. Izadikhah, "An algorithmic method to extend TOPSIS for decision-making problems with interval data," Applied Mathematics and Computation, vol. 175, no. 2, pp. 1375-1384, 2006.

[53] H. M. Moshkovich, A. I. Mechitov, and D. L. Olson, "Ordinal judgments in multiattribute decision analysis," European Journal of Operational Research, vol. 137, no. 3, pp. 625-641, 2002.

[54] Z. S. Xu, Uncertain Multiple Attribute Decision Making: Methods and Applications, Tsinghua Press, Beijing, China, 2005.

[55] N. Bryson and A. Mobolurin, "An action learning evaluation procedure for multiple criteria decision making problems," European Journal of Operational Research, vol. 96, no. 2, pp. 379-386, 1997.

Tao Ding, (1) Liang Liang, (2) Min Yang, (1) and Huaqing Wu (3)

(1) School of Management, Hefei University of Technology, 193 Tunxi Road, Hefei, Anhui 230009, China

(2) Hefei University of Technology, 193 Tunxi Road, Hefei, Anhui 230009, China

(3) School of Economics, Hefei University of Technology, 193 Tunxi Road, Hefei, Anhui 230009, China

Correspondence should be addressed to Min Yang;

Received 27 December 2015; Accepted 31 March 2016

Academic Editor: Victor Shi
TABLE 1: Decision making matrix A of house evaluation.

                [Y.sub.1]   [Y.sub.2]   [Y.sub.3]   [Y.sub.4]

[X.sub.1]          3.0         100         10           7
[X.sub.2]          2.5         80           8           5
[x.sub.3]          1.8         50          20          11
[X.sub.4]          2.2         70          12           9

TABLE 2: Normalized decision making matrix R of house evaluation.

                [Y.sub.1]   [Y.sub.2]   [Y.sub.3]   [Y.sub.4]

[X.sub.1]          0.6         1.0         0.8        0.636
[X.sub.2]         0.72         0.8         1.0        0.455
[x.sub.3]          1.0         0.5         0.4         1.0
[X.sub.4]         0.818        0.7        0.667       0.818

TABLE 3: The cross-evaluation measurements.

Alternatives    [C.sub.max]   [C.sub.1i]   [C.sub.2i]   [C.sub.3i]

[X.sub.1]          0.96          0.96         0.82         0.61
[X.sub.2]          0.92          0.80         0.92         0.66
[X.sub.3]            1           0.52         0.51          1
[X.sub.4]          0.818         0.70         0.70        0.818

Alternatives    [C.sub.4i]   r([X.sub.i])   [delta]([X.sub.i])

[X.sub.1]          0.61          0.75              0.15
[X.sub.2]          0.66          0.76              0.11
[X.sub.3]           1            0.76              0.24
[X.sub.4]         0.818          0.76              0.06

TABLE 4: Alternative efficiency matrix.

Alternatives     [X.sub.l]   [X.sub.2]   [X.sub.3]   [X.sub.4]

CV ([X.sub.i])     0.20        0.14        0.32        0.08

TABLE 5: The results of three methods.

Method              Weight vector         Rank order

Average attribute   (0.25, 0.25,    [X.sub.1] > [X.sub.4] >
value                0.25, 0.25)     [X.sub.2] > [X.sub.3]
Maximizing          (0.20, 0.24,    [X.sub.1] > [X.sub.2] >
deviation            0.29, 0.27)     [X.sub.4] > [X.sub.3]
Information         (0.13, 0.22,    [X.sub.1] > [X.sub.2] >
entropy              0.36, 0.29)     [X.sub.4] > [X.sub.3]

TABLE 6: Attribute weights.

[Y.sub.1]           [Y.sub.2]           [Y.sub.3]

[[w.sup.L.sub.1],   [[w.sup.L.sub.2],   [[w.sup.L.sub.3],
[W.sup.U.sub.1]]    [W.sup.U.sub.2]]    [W.sup.U.sub.3]]

[0.35, 0.50]        [0.15, 0.30]        [0.30, 0.35]

TABLE 7: Attribute values of alternatives.

     [Y.sub.1]      [Y.sub.2]      [Y.sub.3]

A   [0.65, 0.85]   [0.68, 0.86]   [0.45, 0.78]
B     [0.75,1]     [0.55, 0.78]   [0.64, 0.80]
C   [0.54, 0.68]   [0.46, 0.55]     [0.78,1]
D   [0.77, 0.90]   [0.67, 0.78]   [0.56, 0.77]
E   [0.25, 0.45]     [0.89,1]     [0.75, 0.94]

TABLE 8: Evaluation Results for the comparison of faculty candidates.

Candidates   [C.sup.max.sub.i]   [C.sub.1i]   [C.sub.2i]   [C.sub.3i]

A                  0.717           0.717        0.712        0.706
B                  0.792           0.792        0.792        0.788
C                  0.672           0.651        0.661        0.672
D                  0.774           0.774        0.771        0.764
E                  0.645           0.525        0.526        0.540

Candidates   [C.sub.4i]   [C.sub.5i]   [r.sub.i]   [[delta].sub.i]

A              0.717        0.703        0.711          0.006
B              0.792        0.746        0.782          0.018
C              0.651        0.660        0.659          0.008
D              0.774        0.737        0.764          0.014
E              0.525        0.645        0.552          0.047

Candidates   C.V ([X.sub.i])

A                 0.008
B                 0.023
C                 0.012
D                 0.461
E                 0.018

TABLE 9: Ordinal values of alternatives.

            [Y.sub.1]   [Y.sub.2]   [Y.sub.3]   [Y.sub.4]

[X.sub.1]       4           2           4           2
[X.sub.2]       1           4           3           3
[X.sub.3]       2           5           1           1
[X.sub.4]       5           3           2           5
[X.sub.5]       3           1           5           4

TABLE 10: Evaluation results for the selection of community service

Candidates   [C.sup.max.sub.l]   [C.sub.1l]   [C.sub.2l]   [C.sub.3l]

[X.sub.1]          0.489           0.489        0.317        0.295
[X.sub.2]          0.532           0.433        0.532        0.508
[X.sub.3]          0.535           0.526        0.526        0.535
[X.sub.4]          0.223           0.202        0.223        0.211
[X.sub.5]          0.422           0.349        0.349        0.103

Candidates   [C.sub.4l]   [C.sub.5l]   [r.sub.i]   [[delta].sub.i]

[X.sub.1]      0.317        0.317        0.347          0.072
[X.sub.2]      0.458        0.504        0.487          0.036
[X.sub.3]      0.526        0.526        0.528          0.004
[X.sub.4]      0.223        0.223        0.216          0.009
[X.sub.5]      0.349        0.422        0.314          0.109

Candidates   C.V ([X.sub.i])

[X.sub.1]         0.207
[X.sub.2]         0.074
[X.sub.3]         0.008
[X.sub.4]         0.042
[X.sub.5]         0.347
COPYRIGHT 2016 Hindawi Limited
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2016 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:Research Article
Author:Ding, Tao; Liang, Liang; Yang, Min; Wu, Huaqing
Publication:Mathematical Problems in Engineering
Date:Jan 1, 2016
Previous Article:A Nonmonotone Gradient Algorithm for Total Variation Image Denoising Problems.
Next Article:A New Reversible Date-Hiding Algorithm for Encrypted Images.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters