Printer Friendly

Who will test conformance?

When purchasing information technology products conforming to standards, the purchaser should get a known set of functionality, implemented correctly, and tested for conformance to the standard. Increasingly, the marketplace emphasizes conformity to standards. Conformity assessment establishes customer confidence in the products and services they buy.

In many cases, a customer will trust a supplier labeling a product as complying with a standard. In other instances, marketplace or regulatory considerations require the involvement of third party certifiers. In matters relating to health, safety, and the environment, government itself may perform an independent assessment of conformity.

What is the border between an economy in which products conform to standards and an unconstrained economy? Should the government be responsible for assessments of conformance to standards or should private industry be responsible? Unfettered opportunities for business to release new products must balance against product certification the consumer can trust.

Foreign Trade

More and more frequently an industry in one country encounters a requirement to test conformance to a standard in another country before that other country will permit market entry. When these tests duplicate work already performed in the home country, they add to the cost of the product in foreign markets and place the export at a competitive disadvantage.

As a result of the recently concluded international Agreement on Technical Barriers to Trade, no country may impose a technical regulation that is an unjustified barrier to trade. A regulation is a trade barrier, if it is contrary to international standards. Thus, in foreign trade, tests of conformance should only be allowed for international standards.

A country should strive to support its conformance testers so that their tests are accepted in other countries. Conformance testers around the world want to use international guidelines for testing to help their exporters meet the foreign market expectations for certifications and avoid foreign complaints that their countries are operating discriminatory programs.

Government-Related Testing

The International Organization for Standardization (ISO) does not issue certificates of conformity. There is no such thing as "ISO certification" (for information on ISO see Certification of conformance with standards is done independently of ISO by other bodies. ISO does prepare conformity assessment guides. The ISO committee on conformity assessment:

* Prepares international guides relating to the assessment of conformance; and

* Promotes mutual acceptance of national conformity assessment systems.

A very active area internationally concerns the ISO 9000 standards for quality management. The American National Standards Institute (ANSI) wants to help companies get certified as conforming to ISO 9000. ANSI cooperates with the Registrar Accreditation Board to use internationally accepted guidelines in assessing conformance. A primary goal is to assure the user that certification will be accepted in the global marketplace. ANSI provides oversight, public review, rights of appeal, and international liaison for the joint program. The Register Accreditation Board accepts applications, performs the evaluations, and issues the conformance certificates.

The Open Systems Interconnection (OSI) standards from ISO are some of the most important information technology standards. What is the role of the test laboratory and the client during the OSI conformance assessment process? Typically test laboratories are

* Organizations developing or supplying OSI implementations (1st party test labs),

* Organizations willing to verify OSI implementations before using them (2d party test labs), and

* Organizations independent of suppliers or users of OSI implementations whose business is the testing of such implementations (3d party test labs).

Clients may be implementors or suppliers of OSI systems who are applying for their own implementations to be tested, procurers of the implementations, or any interested party.

The American National Institute of Standards and Technology (NIST) has established the GOSIP Testing Program to ensure that networking products purchased by the U.S. government comply with the GOSIP Information Processing Standards. The backbone of GOSIP is OSI. More than 290 products based on GOSIP have been successfully tested and registered. Successfully tested products are entered onto a publicly accessible register.

NIST is collaborating in an international project to automate the development and implementation of test systems for communication protocols. The test system takes as input a conformance test suite based on an international standard. The Telecommunications Laboratory in Taiwan, the French National Institute of Communications, the United Kingdom Department of Trade and Industry, and the Interoperability Technology Association for Information Processing in Japan are among the partners in this project.

The Ada Case

Pioneering conformance testing was supported by the U.S. Department of Defense through its Ada Compiler Validation Capability (ACVC). ACVC represents one of the most comprehensive and successful attempts to test conformance of a software product to an international standard:

* The ACVC was developed under government contract.

* The ACVC test suite was developed alongside the language. Many other language test suites appear only after the development and standardization of the language. The ACVC development identified many bugs in the language that were resolved prior to standardization.

* The test suite is freely available.

* With the wide availability of the free test suite, at the same time as the language standard (and encouraged by Defense Department policy), conformance testing of Ada compilers is the norm, to the point where validation has never been a serious issue for Ada compiler users. Vendors have protested at the costs of validation, but these costs have been controlled, and the net result is that everyone expects an Ada compiler to be validated.

The cost of developing a test suite is significant. Vendors who develop their own test suites (either for internal testing of their products, or as 3d party testers), have an investment that often keeps these tests mostly proprietary. This was part of the problem with Unix testing in the mid-80s, where the AT&T System V Interface Definition was public, but the associated test suite (System V Verification Suite) was controlled by AT&T.

Debate on conformance testing within the Ada community focuses not on "should it be done" but rather on "how much should be done." Should ACVC test all language features? This comprehensiveness would have substantial cost impacts on the vendors for testing conformance. Or should ACVC primarily address "typical user" code and thus provide a test suite that is easier for vendors to run.

The ACVC for Ada83 was criticized for testing too many obscure language ramifications, without providing test coverage on features used by programmers. Compiler vendors invested a large effort in passing the ACVC test suite and had thusly constrained resource to invest in features most important to typical users. Such a result is one of the dangers of government involvement in business.

Non-Governmental Activities

Various professional societies, such as the ACM, have interests in conformance assessment. In the 1980s, the ACM Special Interest Group on Ada (SIGAda) regularly pnblished summaries of the ACVC reports on particular Ada compilers. The ACM Special Interest Group most related to conformance assessment is called SIGMetrics. The April 1995 issue of the ACM SIGMetrics Performance Evaluation Review included an article entitled "TPC Announces New Decision Support Benchmark." The document describes the Transaction Processing Performance Council (TPC) test suite and gives detailed results on the performance of a wide range of hardware platforms on the test suite. TPC is itself a nonprofit corporation whose members include many systems suppliers and database vendors.

Various industry consortia are active in conformance assessment. The Software Publishers Association is a trade association of the Personal Computer software industry and has created the Multimedia PC Working Group (MPCWG). Members of the Executive Board of the MPCWG include AT&T, Fujitsu, IBM, Microsoft, NEC, Olivetti, Philips, and Zenith Data Systems. The MPCWG is responsible for the maintenance of the MPC specification.

End users can be assured software bearing the MPC certification mark is designed to work on a system, upgrade kit, CD-ROM drive, or sound card that meets the MPC specification. Software that is to receive the MPC certification mark must run on the base platform as defined by the MPC hardware specification and must use at least one multimedia element. MPCWG charges an administrative fee of $500 for licensing each software product.

The MPCGW will require hardware that bears the MPC certification mark to pass certification tests provided by the MPCWG. Hardware vendors who want the MPC specification mark may either pay a license fee of $70,000 or royalty fees for full systems of $1 per system sold (for information about MPCWG see MPCWG will argue, as will other private organizations, that the private sector is in the best position to provide cost effective and efficient conformance testing.


Many organizations want conformance assessment. For instance, a U.S. Department of Defense conference on document interchange concluded standards conformance testing was needed for all solutions adopted by the military.

ISO's Long-Range Strategies Group has emphasized the importance of conformance assessment. ISO must ensure that requirements for conformity assessment build worldwide customer confidence and do not in themselves build barriers to trade. The models for how this can be accomplished are still evolving, but mechanisms will be required that give international recognition to conformity to assessment results.

A study mandated by the U.S. Congress and done by the National Research Council said that the U.S. conformity assessment system has become increasingly complex, costly, and burdensome to national welfare (for the executive summary of the study see Unnecessary duplication at the national, state, and local levels results in high costs for U.S. manufacturers, procurement agencies, product certifiers, and consumers. Some recommendations from this study:

* The government should provide NIST with a statutory mandate to implement a policy of phasing out government-operated conformity assessment activities. By the year 2000, the government should rely on private-sector conformity assessment services recognized as competent by NIST.

* The Office of the U.S. Trade Representative (USTR) should continue ongoing mutual recognition agreement negotiations with the European Union. Specifically, the USTR should secure fair access for U.S. exporters to European conformity assessment mechanisms. If such security cannot be obtained within two years, USTR should initiate retaliatory actions as permitted by U.S. trade law.

Should government agencies retain oversight responsibility for critical regulatory and procurement standards in areas of public health, safety, environment, and national security? The answer is certainly that an oversight responsibility in these cases is the government's job. Might the assessment of product conformity to standards be performed most efficiently and effectively by the private sector? The answer to this question might depend on one's political persuasion.

Governmental support of Ada conformance testing has led to more reliable Ada compilers. In general, governments may support standardization work by supporting conformance testing.

As trade becomes more international, the importance of standards and conformance assessment increases. As governments around the world play the "national interest" game, conformance testing becomes one more pawn in the game.

Governments naturally consider communications important and check for compliance to their communications standards. For instance, the U.S. government maintains the Federal Communications Commission (FCC). The FCC inspects radio stations to make sure their equipment and policies are consistent with FCC standards (for Compliance and Information Bureau of the FCC see As information technology and communications merge, how can governments take one approach to communication and a different one to information technology?

The enormous diversity of information technology products and their increasing importance in everyday activities requires that some conformity to standards be guaranteed. Governments have a responsibility to their citizens to oversee the reliability of conformance claims.


The author acknowledges important communications with David Arnold, Anthony Gargaro, and Brian Meek, and would particularly like to point the reader to David Arnold's "Conformance Testing ICT Standards: Costly Care or Valuable Insurance?" in Standardview, 2, 4 (Dec. 1994), 182-187. The bulk of the text in "The Ada Case" section was provided by David Emery.

Roy Rada is a Boeing Distinguished Professor of Software Engineering at Washington State University.
COPYRIGHT 1996 Association for Computing Machinery, Inc.
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1996 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:conformity assessment of information technology products
Author:Rada, Roy
Publication:Communications of the ACM
Date:Jan 1, 1996
Previous Article:The World-Wide Web as social hypertext.
Next Article:Intellectual property rights and the global information economy.

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters