Validation of (Q)SARs models: Jaworska et al.'s response.

We warmly welcome and agree with the comments from Kaiser. In writing the reports that comprised the Mini-Monograph on "Regulatory Acceptance of (Q)SARS" published in EHP and also during the workshop sponsored by the International Council of Chemical Associations and the European Chemical Industry Council and held in Setubal, Portugal, in March 2002 (Jaworska et al. 2003), it was never our stated or underlying intention to make a thorough review of (Q)SARs for toxicologic endpoints and fate. This has been more than adequately achieved in recent editions of Environmental Toxicology and Chemistry (2003) and the Journal of Molecular Structure-THEOCHEM (THEOCHEM 2003). This is firmly stated in the papers themselves (Cronin et al. 2003a, 2003b; Eriksson et al. 2003). Instead, the purpose of these articles was to review the use of (Q)SARs to predict toxicity and fate by regulatory bodies. However, it should be understood, that such reviews will be needed, and are ongoing, in order for (Q)SARs to find a more extensive and transparent use in regulatory schemes and thus achieve their stated goal of reducing animal testing. In general, only predictive techniques used by regulatory bodies are mentioned, although some other work to illustrate various other techniques was cited to provide examples. We did state that the probabilistic neural network (PNN) provided the best predictions for fish toxicity on the basis of an external test set (Cronin et al. 2003a), but like the vast majority of examples of (Q)SARs in the literature, the other endpoints were not included. With regard to the article by Eriksson et al. (2003), the scope was clearly defined in the title; that is, it provided an assessment of regression- and projection-based methods. There are also other approaches to (Q)SAR, such as support vectors machines; however, these were not included because they are not used broadly in all (Q)SAR disciplines. Further, these techniques are not in frequent use by regulatory bodies.

We agree with Kaiser that the performance of a model should be a driving force and is more important than statistics. Appropriate measures of predictivity or performance are enshrined in the cornerstones of current attempts at validation. As part of the validation process, the domain of applicability should also be defined. This will allow a user to know whether a prediction is likely to be valid. It should also be noted that Eriksson et al. (2003) explain that representativity and homogeneity are basic requirements for (Q)SAR modeling, regardless of the statistical method used to develop the model. Eriksson et al. (2003) also noted that these concepts were easy to check and accomplish. Provided the basic conditions of representativity and homogeneity are not violated, statistical descriptors of a model's performance are not misleading. Indeed, failure to comply with representativity and homogeneity will result in the statistical assessment of model performance becoming inappropriate.

The reality is that simple and transparent models are (on the whole) favored by regulatory bodies over more opaque approaches, even if it means losing some statistical fit. We understand that problems could arise in some PNN approaches that have used large numbers of descriptors, some with no relevance to the mechanism of action (Cronin and Schultz 2001). A further issue with PNNs is the fact that they do not provide the same numerical solution to a problem when repeated.

The goal of the Setubal workshop was to encourage further development of the (Q)SAR sciences toward practical use and application. It is important, that as these models develop into more efficient tools for society to understand the fate and effect of chemicals, without extensive use of animals and other resources, they be seen to be valid and suitable for the purpose. To achieve this goal, there are a number of ongoing attempts to validate (Q)SARs (e.g., at the European Union and the Organisation for Economic Co-operation and Development level). These will attempt to weigh all the evidence and produce scientifically sound and usable validation techniques. Interested parties are encouraged to join in these efforts. More information is available on the European Commission Joint Research Centre website (JRC 2003).

The authors declare they have no competing financial interests.

Joanna S. Jaworska

Procter & Gamble Eurocor

Strombeek-Bever, Belgium

E-mail: [e-mail]

Mark Cronin

Liverpool John Moores University 

Liverpool, United Kingdom

Lennart Eriksson

Umetrics

Umea, Sweden

Mike Comber 

ExxonMobil

Brussels, Belgium

REFERENCES

ETC. 2003. Environ Toxicol Chem 22(8):1651-1936.

Cronin M, Schultz TW. 2001, Development of quantitative structure-activity relationships for the toxicity of aromatic compounds to Tetrahymena pyriformis: comparative assessment of the methodologies. Chem Res Toxicol 14:1284-1295.

Cronin MTD, Walker JD, Jaworska JS, Comber MHI, Watts CD, Worth AP. 2003a. Use of QSARs in international decision-making frameworks to predict ecological effects and environmental fate of chemical substances. Environ Health Perspect 111:137-1390.

Cronin MTD, Jaworska JS, Walker JD, Comber MHI, Watts CD, Worth AP. 2003b. Use of QSARs in international decisionmaking frameworks to predict human health effects Environ Health Perspect 111:1391-1401.

Jaworska JS, Comber M, Auer C, Van Leeuwen CJ. 2003, Summary of the workshop on regulatory acceptance of (Q)SARs. Environ Health Perspect 111:135-1360.

JRC. 2003. QSARS, Ispra, Italy:European Commission, Joint Research Centre. Available: http://ecb.jrc.it/QSAR/ [accessed 2 Dec 2003].

Erriksson L, Jaworska J, Worth AP, Cronin MTD, McDowell RM, Gramatica P. 2002. Methods for reliability and uncertainty assessment and for applicability evaluations of classification and regression-based QSARs. Environ Health Perspect 111:1361-1375.

THEOCHEM. 2003. J Mol Structure 629(1-2):1-184.