New hybrid multivariate analysis approach to optimize multiple response surfaces considering correlations in both inputs and outputs/Nova abordagem de analise multivariada hibrido para otimizar multiplas superficies de resposta considerando as correlacoes em ambas as entradas e saidas.
Making decisions about complex problems involving process optimization and engineering design strongly depends on well identified effective factors. From the viewpoint of quality, a process should be designed so that the products could satisfy customer's needs. Quality engineering techniques try to find the interrelations between input parameters and output quality characteristics (also called response variables) as well as to improve outputs.
A common problem in product or process design is to determine optimal level of control variables where there are different outputs, which are often highly correlated. This problem is called multi-response optimization (MRO) with correlated responses.
Several studies have presented approaches addressing multiple quality characteristics but few published papers have focused primarily on the existence of correlation.
Correlation can also meaningfully affect the analysis of MRO problem in another way. Nuisances in experiments may be classified into the following three categories (MONTGOMERY, 2005).
'Known and controllable variables' that are controllable, but their effect is not of interest as a factor. For this kind of nuisance, a technique called blocking can be used to systematically eliminate its effect in the statistical analysis.
'Unknown and uncontrollable variables', that is, the existence of the factor is unknown and it may even be changing levels while the experiments are conducted. Randomization is the design technique used to analyze such a nuisance factor.
'Known and uncontrollable variables', especially, it could be measured during the experiment runs called covariates. In this case, finding individual effect of covariate and their interaction with other variables could help analysts to improve response values.
Complex process or system may be affected by stochastic covariates which can be correlated. The correlation among inputs adds more complexity in estimation as well as optimization.
This paper proposes a methodology that can analyze correlated multiple response surfaces fitted on control factors and correlated covariates. Global criterion (GC) method of vector optimization is also applied since there are several output characteristics to be optimized.
The structure of the remaining part of this paper is as follows. The next section provides a summary of MRO approaches with special focus on correlated responses and correlated covariates. Afterwards, the required information about the proposed methodology is provided. Finally, section 4 illustrates the method by a numerical example.
In multiresponse modeling there are often three types of variables: Factors, nuisances and responses. When a significant degree of correlation exists among the variables, the standard methods cannot estimate the model precisely and, consequently, the optimization results might be unreliable. Modeling and optimization of correlated response surfaces have been recently heightened by many researchers. Chiao and Hamada (2001) considered experiments with correlated multiple responses whose means, variances, and correlations depended on experimental factors. Analysis of these experiments consists of modeling distributional parameters in terms of the experimental factors and finding factor settings which maximize the probability of being in a specification region, i.e., all responses are simultaneously meeting their respective specifications. It is assumed that the multiresponse set has a multivariate normal distribution and also that each response variables is desired to be within a predefined specification region. Kazemzadeh et al. (2008) applied multiobjective goal programming model to provide a general framework for multiresponse optimization problems. Shah et al. (2004) used the seemingly unrelated regressions (SUR) method for estimating the regression parameters where there are correlated dependent variables. The method can be useful in MRS problem with correlated responses and leads to a more precise estimate of the optimum variable setting. PCA is a well-grounded statistical multivariate technique for dimension reduction and making independent components from a set of correlated variables. Tong et al. (2005) used PCA to convert correlated response variables to ordinary response surfaces and also applied a multi-criteria decision-making method called TOPSIS to aggregate several quality characteristics. Antony (2000) used PCA with Taguchi's method. In this method, it is assumed that only those components whose eigenvalues are greater than one can be selected to form final response variables. Thus, their method could not be applied if the problem has more than one component with such characteristic. Tong et al. (2005) determined the optimization direction of each component based on corresponding variation mode charts. Furthermore, Wang (2007) used TOPSIS to find an overall performance index as a criterion for optimizing the multiple quality characteristics.
In order to analyze covariates in MRO problem some research studies have recently been conducted. Hejazi et al. (2011) represented a novel method based on goal programming to find the best combination of factors so as to optimize multiresponse-multicovariate surfaces with consideration of location and dispersion effects. Moreover, they considered covariate probable values as an objective function which should be maximized. Salmasnia et al. (2013) applied PCA to reach uncorrelated sets of responses and covariates. They assumed that the probability distribution functions of the covariates are known. Desirability function was used to aggregate individual desirability of principal components (PCs) extracted from the location and dispersion effects as well as probability of the covariates.
Hejazi et al. (2012) considered correlation coefficients to calculate expected value and variance of goal function in multiresponse optimization problem. They used these measures to construct deterministic equivalent for stochastic optimization models. Hejazi et al. (2013) introduced quality chain design (QCD) problem in multistage systems and proposed a multiresponse optimization model to find best factor setting with smaller covariance matrix. They let the response variables of each stage be considered as covariates affecting responses of the next stages.
A summarized comparison of correlated multiresponse optimization methods are reported in Table 1.
According to the literature, many works have been conducted on using Principal Components Analysis (PCA) to solve correlated multiresponse problems. PCA converts several correlated columns to independent components by linear transformations. These components are then substituted into multiple original responses. Another approach to solve this problem is based on prediction of the correlation as an individual response variable by Response Surface Methodology (RSM). Each of the mentioned approaches has specific benefits and limitations. It seems a sensible claim that PCA cannot provide proper directions for optimization of components. Moreover, if the number of selected components is less than the number of original responses, some information is lost. Consideration of correlation coefficients as separate response variables requires multi-replicated design for experiments. Additionally, the accuracy of estimated correlation is strongly dependent on the number of replications. However, more experiment runs are more costly and time-consuming. Furthermore, even though there are enough experimental runs, the statistical error in response regression is unavoidable. The last approach in solving multiresponse optimization problem is multivariate regression method that is very useful when response variables are correlated.
The proposed method aims to consider all of location effects and correlation among the responses. In addition, probabilistic covariates are included into the multiresponse model to reduce error terms and uncovered variance.
Material and methods
When the problems involve several equations with common variables, it is recommended to estimate the parameters through a system of equations simultaneously. Various methods such as Ordinary Least Squares (OLS), Cross-Equation Weighting method, SUR, Two-Stage Least Squares (2SLS), Weighted Two-Stage Least Squares (WTSLS), Three-stage Least Squares (3SLS), Full Information Maximum Likelihood (FIML), and the Generalized Method of Moments (GMM) have been proposed to solve such problems. Among them, SUR and FIML methods have been used in this paper to estimate the response surfaces simultaneously.
The SUR method, also known as the multivariate regression, or Zellner's method, estimates the parameters of the system, accounting for heteroscedasticity and contemporaneous correlation in the errors across equations.
Full Information Maximum Likelihood (FIML) estimates the likelihood function under the assumption that the contemporaneous errors have a joint normal distribution.
The aforementioned methods are compared with respect to the main characteristics in Table 2.
In this study, there are two main approaches included in the proposed methodology to analyze correlation among the inputs as well as the outputs. The covariates are initially transformed by PCA to remove their correlation and after that, the response surfaces between correlated response variables and input (including PCs and control factors) are fitted through a simultaneous equations system.
Consecutive steps of the proposed approach are as follows:
Step 1: Identify input and outputs variables.
In this step, all potentially effective variables (namely responses, factors, covariates and other nuisances) should be identified.
Step 2: Select a proper design and run the experiments.
A proper design is selected for conducting the experiments regarding the number of variables and their levels.
Step 3: PCA phase.
Perform PCA on correlated covariates to get independent components (see appendix (A) for more details about PCA).
Step 4: Develop a system of equations.
4) a. Perform an initial RSM to get an insight about the more effective factors on each response.
4) b. Define an equation for relations between each response and other variables.
Next, enter each response variable and related factors as an equation into the system. In addition let each response be considered as a predictor variable for other ones.
Step 5: Estimate parameters of the system.
If the error terms are normally distributed, use FIML, otherwise perform ISUR method to estimate the coefficient of effects.
Step 6: Construct multi-objective optimization model including the following objective functions.
--Response surfaces related to quality characteristics.
--Probability function of the PCs derived by using PCA transformation equations and probability function of original covariates.
Step 7: Apply Global Criterion (GC) method to solve the multi-objective optimization model.
In Section 4 these steps are discussed in details.
A general multiresponse problem can be
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1)
Subject to: 1 < x < u ; lcl < c < ucl
[[??].sub.i] (x) represents response surface for ith quality characteristic;
f ([Pc.sub.j]) is the probability function of jth PC;
x is vector of control factors;
c is covariate vector calculated by inverting the
Furthermore, it is assumed that the process is statistically under control and the control range for covariate vector is [lcl, ucl].
Optimization method (Global Criterion)
This method allows one to transform a multio-bjective optimization problem into a single-objective problem. The function traditionally used in this method is distance. The multi-objective method can be written as follows:
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (2)
where [T.sub.i] is the optimum value of problem objective function when only ith objective was considered; wi is a value representing importance of each objective; di is the range of ith response within the observed experimental runs (DONOSO; FABREGAT, 2007). In this study GC method was applied to convert problem into single objective form.
Results and discussion
This section is organized to demonstrate the computational steps of the proposed approach. For this purpose, a numerical example from the literature is considered with some modifications (MONTGOMERY, 2005).
Step 1: A chemical experiment with three controllable variables and two covariates is designed to be analyzed by the proposed method. The outputs are conversion (Y1) and activity (Y2) levels. Humidity (c1) and environment temperature (c2) are considered as probabilistic covariates.
Step 2: A CCD design is selected and the experiments are conducted accordingly. Table 3 shows the results of experiments gathered by a Central Composite Design (CCD).
Step 3. PCA is performed on Humidity and Temperature factors. According to the observations, they have the following probability distribution.
Since, there is a significant linear relationship between two covariates, it is reasonable to consider a bivariate distribution for their treatments. It may be observed that these two covariates follow a normal distribution with the following parameters:
[c.sub.1] ~ N(0.5032, [(0.2278).sup.2]), [c.sub.2] ~ N(15.30, [(2.581).sup.2]), and [rho]([c.sub.1], [c.sub.2]) = 0.655 (3)
Consider the above distributions as marginal probability functions of c1 and c2. Therefore, the bivariate normal probability distribution for the covariates can be estimated as follows:
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (4)
PCA gives the following equations to transform the set of covariates into a set of independent ones (The required calculations are performed in Minitab statistical package).
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (5)
Step 4. Understanding the strong effects helps us to fit better surfaces of response variables. Therefore, Figure 1 is provided to show the effects graphically and separate RSMs have been initially conducted on each response to guess which predictive terms should be included in the estimation. The results showed that the following terms would be considered to construct the system of equations.
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]
In this case, the problem is analyzed by Iterative Seemingly Unrelated Regression (ISUR) and FIML. The response surfaces regressed by the mentioned methods are given below in Table 4 (Eviews statistical package has been used to estimate the parameters in system).
Step 5. Construct the multi objective optimization model
Two response surfaces and two probability functions are to be considered as objective functions with respect to input variables constrained by their specification limits. Therefore, the multi-objective mathematical program for this problem is developed in which the decision variables consist of three factors and two interdependent covariates.
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (6)
Subject to: [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]
The last constraints calculate the original value of covariates by inverting the transformation matrix (A) and ensure that the covariates are within the pre-specified statistical control limits. The following calculations are required to calculate the probability function of the PCs.
Theorem 1- If C is vector of p random variables jointly distributed by [N.sub.p]([[mu].sub.c], [[SIGMA].sub.c]), and A isa q x p matrix, then the distribution of PC = AC remains a multivariate normal with the following parameters (Proofs are available in Rencher and Schaalje (2008)).
[[mu].sub.PC] = A[[mu].sub.c] (7)
[[SIGMA].sub.PC] = A[[SIGMA].sub.C]A' (8)
where A' is the transpose of matrix A.
According to Theorem 1, the distribution function of the PCs is given below.
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]
As shown above, the new components have zero covariance so their probability distributions can be expressed by two individual and univariate normal variables.
pc1~N(15.3, 6.682) and pc2~N(-0.4, 0.029)
Now, model represented by Equation set (6) can be explicitly formed as:
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (9)
The same constraints
Model (9) is a nonlinear programming due to the first two objective functions. It can be simplified to quadratic programming model by considering this point that the mode value of each normal distribution occurs at mean value. Therefore, the maximum probability equals to minimum distance form mean value.
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (10)
With this property of normal distribution, the final multiobjective quadratic programming can be written as:
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (11) Subject to:
The same constraints
Table 5 gives a summary of optimal solutions obtained by solving the above model for each objective functions separately.
According to Table 6, the final multi-objective mathematical model using Global Criterion can be constructed by replacing the objective functions of the above multi-objective program as Equation (6).
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (12)
In this example, we consider the same important degrees for all objective functions. Table 6 shows the optimal solution and the related objective values for this example.
The results support the claim that the method which applies PCA on outputs cannot correctly find optimization direction. But the application of PCA to solve co-linearity among covariates would lead into better and more accurate estimations. It is also observed that most probable values of covariates would lead into the more reliable results. The PCA method reaches the target of first objective due to the large coefficient of first response in the first PC. It seems PCA is more useful for correlated predictors rather than correlated multiresponse problems. Most existing MRO works used PCA to gain uncorrelated responses, but they usually disregarded the proper direction of location effects. Moreover, the proposed methodology has following main features:
Table 6. Optimal results of the numerical example. Method X PC C ISUR [MATHEMATICAL [MATHEMATICAL [MATHEMATICAL EXPRESSION NOT EXPRESSION NOT EXPRESSION NOT REPRODUCIBLE IN REPRODUCIBLE IN REPRODUCIBLE IN ASCII] ASCII] ASCII] FIML [MATHEMATICAL [MATHEMATICAL [MATHEMATICAL EXPRESSION NOT EXPRESSION NOT EXPRESSION NOT REPRODUCIBLE IN REPRODUCIBLE IN REPRODUCIBLE IN ASCII] ASCII] ASCII] (RIBEIRO et [MATHEMATICAL Not [MATHEMATICAL al., 2010) EXPRESSION NOT considered EXPRESSION NOT REPRODUCIBLE IN REPRODUCIBLE IN ASCII] ASCII] Method [R.sub.1](x,c,y) [R.sub.2](x,c,y) GC ISUR 100 70.837 0.0018 FIML 100 78 0.0522 (RIBEIRO et 100 62.62 Not al., 2010) considered
Multiple responses, multiple stochastic covariates have been analyzed by the methodology, The effects of covariates with known distribution function can be identified in this approach,
PCA is used to solve co-linearity issues when there are meaningful dependencies among the covariates.
Several objective functions and performance indices of a quality engineering problem can be optimized simultaneously by using GC method,
The desired direction for optimization of responses doesn't change after modeling and optimization.
This study proposes a new hybrid approach on multiresponse optimization in which PCA method applies to handle co-linearity among the covariates and uses multivariate system regression to predict the correlated responses. Current study tries to model the multiresponse-multicovariate problem in a simultaneous system of equations and use the estimated equations to construct an optimization program.
For further studies, the mixed set of categorical and numerical responses is suggested. In this work, only the variances of observed values were considered. Therefore, the variances of predicted responses can be another future research on this subject.
Principal component analysis (PCA)
Principal Component Analysis was initially introduced by Pearson (1901) and later developed by Hotelling (1933). PCA is a multivariate technique for dimension reduction and forming independent components from correlated variables. The maximum number of new variables that can be formed is equal to the number of original variables. If we have a set of p correlated variables, PCA generates p uncorrelated ones by linear combinations as follow:
[pc.sub.1] = [w.sub.11] [x.sub.1] + [w.sub.12] [x.sub.2] + ... + [w.sub.1p] [x.sub.p] (A1)
[Pc.sub.2] = [w.sub.21] [x.sub.1] + [w.sub.22] [x.sub.2] + ... + [w.sub.2p] [x.sub.p] (A2)
[Pc.sub.p] = [w.sub.p1] [X.sub.1] + [w.sub.p2] [x.sub.2] + ... + [w.sub.pp] [X.sub.p] (A3)
where [p.sub.c1], [p.sub.c2], ..., [p.sub.cp] are the p principal components and [w.sub.ij] is the weight of the jth variable for the ith principal component. The weights, [w.sub.ij], are estimated such that:
1 The principal components are created in order to decreasing variance, and therefore the first principal component accounts for most variance in the data.
Second component is found so that it can cover maximum amount of the variance which is not identified by the first one and so on.
2 [w.sub.ij.sup.2]+[w.sub.i2.sup.2] + ... + [w.sub.ip.sup.2] = 1 i = 1, 2, ..., p (A4)
3 [w.sub.i1][w.sub.j1] + [w.sub.i2][w.sub.j2] + ... + [w.sub.ip][w.sub.jp] = 0 for all i [not equal to] j (A5)
Condition (2) is used to fix the scale of the new variables and is necessary because it is possible to increase the variance of a linear combination by changing the scale of the weights. The condition (3) ensures that W=[([w.sub.ij]).sub.pxp] is an orthogonal matrix (Sharma, (1995)).
Taha Hossein Hejazi, Mirmehdi Seyyed-Esfahani * and Majid Ramezani
Department of Industrial Engineering and Management Systems, Amirkabir University of Technology, Tehran, Iran. *Author for correspondence. E-mail: firstname.lastname@example.org
AMEMIYA, T. The maximum likelihood and the nonlinear three-stage least squares estimator in the general nonlinear simultaneous equation model. Econometrica, v. 45, n. 4, p. 955-968, 1977.
ANTONY, J. Multi-response optimization in industrial experiments using Taguchi's quality loss function and principal component analysis. Quality and Reliability Engineering International, v. 16, n. 1, p. 3-8, 2000.
BASMANN, R. L. A generalized classical method of linear estimation of coefficients in a structural equation. Econometrica, v. 25, n. 1, p. 77-83, 1957.
CHIAO, C. H.; HAMADA, M. Analyzing experiments with correlated multiple responses. Journal of Quality Technology, v. 33, n. 4, p. 451-465, 2001.
DONOSO, Y.; FABREGAT, R. Multi-objective optimization in computer networks using metaheuristics. Boca Raton: Auerbach Publications, 2007.
HANSEN, L. P. Large sample properties of generalized method of moments estimators. Econometrica, v. 50, n. 4, p. 1029-1054, 1982.
HEJAZI, T.; BASHIRI, M.; NOGHONDARIAN, K.; ATKINSON, A. C. Multiresponse optimization with consideration of probabilistic covariates. Quality and Reliability Engineering International, v. 27, n. 4, p. 437-449, 2011.
HEJAZI, T. H.; BASHIRI, M.; DIAZ-GARCIA, J. A.; NOGHONDARIAN, K. Optimization of probabilistic multiple response surfaces. Applied Mathematical Modelling, v. 36, n. 3, p. 1275-1285, 2012.
HEJAZI, T. H.; SEYYED-ESFAHANI, M.; MAHOOTCHI, M. Quality chain design and optimization by multiple response surface methodology. The International Journal of Advanced Manufacturing Technology, v. 68, n. 1-4, p. 1-15, 2013.
HOTELLING, H. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, v. 24, n. 6, p. 417, 1933.
KAZEMZADEH, R. B.; BASHIRI, M.; ATKINSON, A. C.; NOOROSSANA, R. A general framework for multiresponse optimization problems based on goal programming. European Journal of Operational Research, v. 189, n. 2, p. 421-429, 2008.
MONTGOMERY, D. C. Design and analysis of experiments. 6th ed. Hoboken: John Wiley and Sons, 2005.
PEARSON, K. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, v. 2, n. 11, p. 559-572, 1901.
RENCHER, A. C.; SCHAALJE, G. B. Linear Models in Statistics. 2nd ed. New York: Wiley-Interscience, 2008.
RIBEIRO, J. S.; TEOFILO, R. F.; AUGUSTO, F.; FERREIRA, M. M. C. Simultaneous optimization of the microextraction of coffee volatiles using response surface methodology and principal component analysis. Chemometrics and Intelligent Laboratory Systems, v. 102, n. 1, p. 45-52, 2010.
SALMASNIA, A.; BARADARAN KAZEMZADEH, R.; SEYYED-ESFAHANI, M.; HEJAZI, T. H. Multiple response surface optimization with correlated data. The International Journal of Advanced Manufacturing Technology, v. 64, n. 5-8, p. 841-855, 2013.
SHAH, H. K.; MONTGOMERY, D. C.; CARLYLE, W. M. Response surface modeling and optimization in multiresponse experiments using seemingly unrelated regressions. Quality Engineering, v. 16, n. 3, p. 387-397, 2004.
SHARMA, S. Applied multivariate techniques. Hoboken: John Wiley and Sons, 1995.
SU, C. T.; TONG, L. I. Multi-response robust design by principal component analysis. Total Quality Management, v. 8, n. 6, p. 409-416, 1997.
TONG, L. I.; WANG, C. H.; CHEN, H. C. Optimization of multiple responses using principal component analysis and technique for order preference by similarity to ideal solution. International Journal of Advanced Manufacturing Technology, v. 27, n. 3-4, p. 407-414, 2005.
WANG, C. H. Dynamic multi-response optimization using principal component analysis and multiple criteria evaluation of the grey relation model. International Journal of Advanced Manufacturing Technology, v. 32, n. 5-6, p. 617-624, 2007.
ZELLNER, A. An efficient method of estimating seemingly unrelated regressions and tests for aggregation bias. Journal of the American Statistical Association, v. 57, n. 298, p. 348-368, 1962.
ZELLNER, A.; THEIL, H. Three-stage least squares: simultaneous estimation of simultaneous equations. Econometrica, v. 30, n. 1, p. 54-78, 1962.
Received on June 8, 2012.
Accepted on October 6, 2013.
License information: This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Table 1. Comparative study of the major works on MRO with correlated data. Method Solution Location Dispersion Space Effect Effect (KAZEMZADEH Continuous * * et al., 2008) (SHAH et al., 2004) Continuous * (sU; TONG, 1997) Discrete * (tong et al., 2005) Discrete * (CHIAO; HAMADA, 2001) Continuous * * (ANTONY, 2000) Discrete * (WANG, 2007) Discrete * * (RIBEIRO et al., 2010) Continuous * (HEJAZI et al., 2011) Continuous * (SALMASNIA et al., 2013) Continuous * * (HEJAZI et al., 2012) Continuous * * (HEJAZI et al., 2013) Continuous * * Method Interaction Approach on Effect correlated inputs (KAZEMZADEH * Not considered et al., 2008) (SHAH et al., 2004) * Not considered (sU; TONG, 1997) Not considered (tong et al., 2005) Not considered (CHIAO; HAMADA, 2001) * Not considered (ANTONY, 2000) Not considered (WANG, 2007) Not considered (RIBEIRO et al., 2010) * Not considered (HEJAZI et al., 2011) * PCA (SALMASNIA et al., 2013) * PCA (HEJAZI et al., 2012) * Not-considered (HEJAZI et al., 2013) * Simulation Method Approach on correlated outputs (KAZEMZADEH Considers Correlation et al., 2008) Coefficient as responses (SHAH et al., 2004) SUR method (sU; TONG, 1997) PCA (tong et al., 2005) PCA (CHIAO; HAMADA, 2001) Considers Correlation Coefficient as responses (ANTONY, 2000) PCA (WANG, 2007) PCA (RIBEIRO et al., 2010) PCA (HEJAZI et al., 2011) Not considered (SALMASNIA et al., 2013) PCA (HEJAZI et al., 2012) Considered in calculating variance of the Goal function (HEJAZI et al., 2013) Iterative SUR Method Optimization Approach (KAZEMZADEH Goal programming and et al., 2008) Desirability fucntion (SHAH et al., 2004) Desirability function (sU; TONG, 1997) Factor effects of new components (tong et al., 2005) Variation mode chart of PCA (CHIAO; HAMADA, 2001) Joint probability maximization (ANTONY, 2000) Signal to noise maximization (WANG, 2007) Variation mode chart of PCA (RIBEIRO et al., 2010) Response surface fitting on first component (HEJAZI et al., 2011) Goal programming (SALMASNIA et al., 2013) Desirability function (HEJAZI et al., 2012) Goal programming (HEJAZI et al., 2013) Minimizing the determinant and trace of the predicted covariance matrix Table 2. Characteristics of the major methods of system estimation. Method of estimation Limiting assumptions Normality Homoscedasticity OLS -- * Cross-Equation -- - Weighting sur -- - (ZELLNER, 1962) 2SLS -- * (BASMANN, 1957) WTSLS -- -- 3SLS -- -- (ZELLNER; THEIL, 1962) FIML * -- (AMEMIYA, 1977) GMM -- -- (HANSEN, 1982) Method of estimation Limiting assumptions IPE (1) IET (2) Instruments OLS * * No Cross-Equation * * No Weighting sur -- -- No (ZELLNER, 1962) 2SLS -- * Yes (BASMANN, 1957) WTSLS -- * Yes 3SLS -- -- Yes (ZELLNER; THEIL, 1962) FIML - -- No (AMEMIYA, 1977) GMM * -- Yes (HANSEN, 1982) (1)-Independency between Predictors and Errors (2)-Independent error terms. Table 3. Results of designed experiments for numerical example. Time Heat Catalyst Humidity Temp ([x.sub.1]) ([x.sub.2]) ([x.sub.3]) ([c.sub.1]) ([c.sub.2]) -1.000 -1.000 -1.000 41% 16.7 1.000 -1.000 -1.000 55% 17.3 -1.000 1.000 -1.000 67% 19.3 1.000 1.000 -1.000 55% 12.3 -1.000 -1.000 1.000 12% 11.5 1.000 -1.000 1.000 95% 18.5 -1.000 1.000 1.000 65% 19.2 1.000 1.000 1.000 96% 16.5 0.000 0.000 0.000 30% 13.2 0.000 0.000 0.000 59% 14.0 0.000 0.000 0.000 46% 16.4 0.000 0.000 0.000 57% 16.4 -1.682 0.000 0.000 59% 13.5 1.682 0.000 0.000 33% 13.9 0.000 -1.682 0.000 48% 15.0 0.000 1.682 0.000 38% 13.1 0.000 0.000 -1.682 29% 12.7 0.000 0.000 1.682 20% 15.8 0.000 0.000 0.000 25% 11.5 0.000 0.000 0.000 75% 19.1 Time [pc.sub.1] [pc.sub.2] Conversion Activity ([x.sub.1]) ([R.sub.1]) ([R.sub.2]) -1.000 16.719 -0.572 74.000 53.200 1.000 17.298 -0.469 51.000 62.900 -1.000 19.284 -0.471 88.000 53.400 1.000 12.327 -0.171 70.000 62.600 -1.000 11.467 -0.561 71.000 57.300 1.000 18.486 -0.140 90.000 67.900 -1.000 19.220 -0.482 66.000 59.800 1.000 16.528 -0.015 97.000 67.800 0.000 13.243 -0.481 81.000 59.200 0.000 13.973 -0.233 75.000 60.400 0.000 16.432 -0.505 76.000 59.100 0.000 16.377 -0.397 83.000 60.600 -1.682 13.494 -0.200 76.000 59.100 1.682 13.889 -0.485 79.000 65.900 0.000 15.024 -0.401 85.000 60.000 0.000 13.098 -0.389 97.000 60.700 0.000 12.707 -0.459 55.000 57.400 0.000 15.831 -0.731 81.000 63.200 0.000 11.530 -0.432 80.000 60.800 0.000 19.142 -0.378 91.000 58.900 Table 4. Estimated equations in the system using FIML and ISUR method. Method Estimated system ISUR R1(X,PC,Y) = 79.6+1.028 [x.sub.1]+3.898 [x.su.2]+6.203 [x.su.3]+ + 11.481 [x.sub.1][x.sub.3]-3.901 [x.sub.2][x.sub.3]+3.103[x.sup.2.sub.2]- 5.012[x.sub.3.sup.2] R2(X,PC,Y)= 43.544 + 0.928 [x.sub.1]+2.37 [x.sub.3]+1.37 [pc.sub.1]+10.066 [pc.sub.2]+ + 0.267 Y1-5.868 [x.sub.1] [pc.sub.2]-0.0177 [pc.sub.1] [Y.sub.1]+0.97[x.sub.3.sup.2] FIML R1(X,PC,Y)= 79.6+1.028 [x.sub.1]+3.925 [x.sub.2]+6.204 [x.sub.3]+ + 11.481 [x.sub.1][x.sub.3]-4.007 [x.sub.2][x.sub.3] +3.021 [x.sub.2.sup.2] 2-5.0198[x.sub.3.sup.2] R2(X,PC,Y)= 23.33+0.889 [x.sub.1]+2.17 [x.sub.3]+2.595 [pc.sub.1]+10.859 [pc.sub.2]+ + 0.531 [Y.sub.1]-5.811 [x.sub.1] [pc.sub.2]-0.033 [pc.sub.1] [Y.sub.1]+1.287[x.sub.3.sup.2] Table 5. Trade off matrix and required parameters of GC method. Method of [Z.sub.1] [Z.sub.2] [R.sub.1] [R.sub.2] estimation Target ISUR 0 0 100 73.832 (Ti) FIML 0 0 100 78.796 Best 0.106 0.003 97 67.9 observed Worst 1.541 2.272 51 53.2 observed Range 1.435 2.269 46 14.7 (di)