An overview of method validation--part 2.

(Continued from the December 2003 issue)

Using validation data to design QC

Quality control and quality assurance are terms whose meanings are often varied according to the context. In practical terms, quality assurance relates to the overall measures taken by a laboratory to ensure and regulate quality, whereas quality control describes the individual measures which relates to the monitoring and control of particular analytical operations.

Method validation gives an idea of a method's performance capabilities and limitations, which may be experienced in routine use while the method is in control. In routine use, specific controls need to be applied to the method to verify that it remains in control, i.e., is performing in the way expected. During the validation stage, the method was largely applied to samples of known content. Once the method is in routine use, it is used for samples of unknown content. Suitable control can be applied by continuing to measure samples of known content, thus allowing the analyst to decide whether the variety of answers obtained truly reflects the diversity of the samples analyzed or whether unexpected and unwanted changes are occurring in the method performance. In practice, these known samples should be measured with every batch of samples as part of the quality control process.

The sort of checks made will depend on the nature, criticality and frequency of the analysis, batch size, degree of automation and test difficulty, and also on the lessons learned during development and validation processes. Quality control can take a variety of forms, both inside the laboratory (internal) and between the laboratory and other laboratories (external).

Internal QC

This includes the use of blanks, chemical calibrants, spiked samples, blind samples, replicate analysis and QC samples. The use of control charts is recommended, particularly for monitoring results from QC control samples.

The sorts of QC adopted must be demonstrably sufficient to ensure the validity of the results. Different sorts of quality control may be used to monitor different types of variation within the process. QC samples, analyzed at intervals in the analytical batch, will indicate drift in the system; use of various types of blanks will indicate contributions to the instrument signal besides those from the analyte; and duplicate analyses give a check of repeatability.

QC samples are typical samples which over a given period of time are sufficiently stable and homogeneous to give the same result and available in sufficient quantities as to be available for repetitive analysis. Over this period, the random variation in performance of the analytical method can be monitored by monitoring the analyzed value of the QC sample, usually by plotting it on a control chart. Limits are set for the values on the chart (conventionally, warning limits are set at [+ or -] 2[sigma] ([+ or -] 2s) about the mean value, and action limits are set at [+ or -] 3[sigma] ([+ or -] 3s) about the mean value). As long as the QC sample value is acceptable, it is likely that results from samples in the same batch as the QC sample can be taken as reliable. The acceptability of the value obtained with the QC sample should be verified as early as practicable in the analytical process so that in the event of a problem, as little effort as possible has been wasted on unreliable analysis of the samples themselves.

In order to set realistic limits on the control chart, the initial calculations of mean and standard deviation must reflect the way the method is actually intended to be used on a day to day basis. Thus, readings should mimic all possible variations in operating conditions, different analysts, variations in laboratory temperature, etc. If this is not done, then the standard deviation will be unrealistically small, resulting in limits being set on the chart, which cannot possibly be compiled within normal use.

The use of various types of blanks enables the analyst to ensure that calculations made for the analyte can be suitably corrected to remove any contributions to the response which are not attributable to the analyte. Replicate analysis provides a means of checking for changes in precision in an analytical process, which could adversely affect the result. Replicates can be adjacent in a batch (to check repeatability) or placed randomly (to check for drift).

Blind analysis is effectively a form of repeat analysis and provides a means of checking precision. It consists of replicated test portions placed in the analytical batch, possibly by the laboratory supervisor, and is so-called because the analyst is not normally aware of the identity of the test portions or that they are replicates. Thus, the analyst has no preconceived ideas that the particular results should be related.

Standards and chemical calibrants placed at intervals in an analytical batch enable checks to be made that the response of the analytical process to the analyte is stable. It is the responsibility of the laboratory management to set and justify an appropriate level of quality control, based on risk assessment, taking into account the reliability of the method, the criticality of the work and the feasibility of repeating the analysis if it does not work correctly the first time. It is widely accepted that for routine analysis, a level of internal QC of 5% has been identified as reasonable, i.e., one in every 20 samples analyzed should be a QC sample.

However, for robust, routine methods with high sample throughput, a lower level of QC may be reasonable. For more complex procedures, a level of 20% is not unusual, and on occasions even 50% may be required. For analyses performed infrequently, a full system validation should be performed on each occasion. This may typically involve the use of a reference material containing a certified or known concentration of analyte, followed by replicate analyses of the sample and spiked sample (a sample to which a known amount of the analyte has been deliberately added). Those analyses undertaken more frequently should be subject to systematic QC procedures incorporating the use of control charts and check samples.

External QC

A recognized way for a laboratory to monitor its performance against both its own requirements and the norm of peer laboratories is through regular participation in regular proficiency testing schemes. Proficiency testing helps to highlight reproducibility performance between laboratories and systematic errors, i.e., bias. It can also be used to determine repeatability, but this can also be checked more cost effectively using internal controls. Proficiency testing and other types of inter-comparison are accepted as being an important means of monitoring traceability at national and international levels. Accreditation bodies recognize the benefit of these schemes and strongly encourage laboratories to participate in proficiency testing as an integral part of their quality assurance protocols. It is important to monitor proficiency testing results as a means of checking quality assurance, and take action as necessary. In certain instances, accreditation bodies may specify participation in a particular proficiency-testing scheme as a requirement of accreditation. The value of proficiency testing is, of course, only as good as the schemes themselves. Very often there may not be a scheme available which is relevant to the types of analysis that the laboratory wishes to check, especially if it is working in isolation.

Implication of validation data for calculating results and reporting

It is important that the analyst is able to translate the data generated during analysis of samples using the validated method into answers, which directly relate to solving the customer's problem. The performance characteristics established during the validation process help to do this. Precision data for repeatability and reproducibility can be used to establish whether differences found while analyzing samples are significant. Quality controls based on the validation data can be used to confirm that the method is in control and producing meaningful results. Estimation of the measurement uncertainty, associated with the method performance, enables expression of the result as a range of values in which the true value for the measurement can be said to lie with an accepted level of confidence.

It is important that the analyst has accessed the validation data, which can be used to support the validity of the results. Issues such as method validation, variability and measurement uncertainty need to be treated carefully in certain circumstances; for example, in legal or forensic contexts. It may be better to be open about the existence of uncertainty attached to measurements and be prepared to justify decisions made in the light of knowing that uncertainty.

When reporting results, the analyst must decide whether to correct for any biases which may have been detected, or to report results uncorrected, but acknowledge the existence of the bias.

Care should be taken when reporting results as "not detected." On its own, this statement is uninformative and should be accompanied by an explanation of what is the limit of detection in that instance. Sometimes it is appropriate to report a numerical value, even though this may be below the apparent limit of detection.

As has been explained above, much of the information required to evaluate the uncertainty is obtainable from the validation process. It is assumed that before the method is used on unknown samples, the laboratory will have demonstrated that it can achieve the performance parameters set out in the method, and that it has satisfactory QC data and satisfactory results from measurements on reference materials.

Where a statement of uncertainty is required with results, it may be appropriate to quote an expanded uncertainty by applying a suitable coverage factor, e.g., a coverage factor of 2, approximates to 95% confidence.

Validation techniques at a glance

What to analyze          What to calculate        Comments

Confirmation of identity and selectivity/specificity

* Analyze samples, and   * Use the results from   * Decide how much
  reference materials.     the confirmatory         supporting evidence
* Analyze samples          techniques to assess     is reasonably
  containing various       the ability of the       required to give
  suspected                method to confirm        sufficient
  interferences in the     analyte identity and     reliability.
  presence of the          its ability to         * If detection or
  analyte of interest.     measure the analyte      quantitation is
                           in isolation from        inhibited by the
                           other interference.      interference,
                         * Examine effect of        further method
                           interference.            development will
                                                    be required.

Limit of detection (LoD)

* Ten independent        * Sample standard        * This is only
  sample blanks            deviation 's' of         useful where the
  measured once each.      sample blank values      sample blank gives
* Ten independent          or fortified sample      a non-zero standard
  sample blanks            blank values.            deviation.
  fortified at lowest    * Express LoD as
  acceptable               the analyte
  concentration            concentration
  measured once each.      corresponding to
* Sample blanks spiked     mean sample blank
  with the analyte         value +3s or 0 +3s
  at a range of            for fortified
  concentration            samples.
  levels. At each        * A response curve of
  concentration level,     % positive (or
  it will be necessary     negative) results
  to measure               vs. concentration
  approximately ten        should be
  independent              constructed, from
  replicates randomly.     which it will be
                           possible to
                           determine, by
                           inspection, the
                           threshold
                           concentration at
                           which the test
                           becomes unreliable.

Limit of quantitation (LoQ)

* Ten independent        * Sample standard        * If measurements
  sample blanks            deviation 's' of         are made under
  measured once each.      sample blank value.      repeatability
* Fortify aliquots of      Express LoQ as           conditions, a
  a sample blank at        the analyte              measure of the
  various analyte          concentration            repeatability
  concentrations close     corresponding to the     precision at this
  to the LoD. Measure      sample blank value       concentration is
  once each, ten           plus either 5s, 6s       also obtained.
  independent              or 10s.
  replicates at each     * Calculate the
  concentration level.     standard deviation
                           's' of the analyte
                           value at each
                           concentration.
                           Plots against each
                           concentration and
                           put assigned a value
                           to the LoQ by
                           inspection. Express
                           LoQ as the lowest
                           analyte
                           concentration, which
                           can be determined at
                           an acceptable level
                           of uncertainty.

Working and linear range

* Blank plus reference   * Plot measurement       * Ideally the
  materials or             response (y-axis)        different
  fortified sample         against measured         concentrations are
  blanks at various        concentration            to be prepared
  concentration (need      (x-axis).                independently.
  at least six           * Calculate              * It is unsafe to
  concentration plus       appropriate              remove outliers
  blank).                  regression co            without first
* Reference materials      efficient. Calculate     checking using
  or fortified sample      and plot the             further
  blanks at least six      residual values          determination at
  different                (difference between      nearby
  concentrations           actual y-value and       concentrations.
  within the linear        the y-value            * Use a weighted
  range.                   predicted by the         regression
                           straight line, for       calculation
                           each x-value).           if variance of
                           Random distribution      replicates is
                           about the straight       proportional to
                           line confirms the        concentrations.
                           linearity.
                           Systematic trends
                           indicate
                           non-linearity.

Accuracy and trueness

* Reagent blank and      * Mean blank value       * Independent method
  reference material       substracted from         may have biases
  using candidate          mean analyte value       of its own, hence
  method.                  for reference            not an absolute
* Reagent blank and        material. Compare        measure of
  reference/test           with true or             accuracy.
  material using           accepted true value    * Primary method
  candidate method and     for the reference        ideally has no
  independent method.      material which gives     biases, so is a
                           a measure of the         better measure of
                           method's bias.           accuracy.

Repeatability and reproducibility precision

Standards, reference     * Determine standard     * Determines
  materials or             deviation at each        repeatability
  fortified sample         concentration for        standard deviation
  blanks at various        all cases.               at each
  concentration across                              concentration.
  a working range.                                * Determine
* Same analyst,                                     intralaboratory
  equipment,                                        reproducibility
  laboratory, short                                 standard deviation
  time scale.                                       at each
* Different analyst,                                concentration.
  equipment,                                      * Determines
  laboratory, short                                 interlaboratory
  time scale.                                       reproducibility
* Different analysts,                               standard deviation
  equipment,                                        at each
  laboratories,                                     concentration.
  extended time scale.

Ruggedness or robustness

* Identify variables     * Determine the effect   * Design quality
  which could have a       of each change           control in order
  significant effect       of condition on the      to control the
  on method                mean.                    critical
  performance.           * Rank the variables       variables.
* Set up experiments       in order of the
  to monitor the           greatest effect on
  effects on accuracy      method performance,
  and precision of
  systematically
  changing the
  variables.

Recovery

* Matrix blanks or       * Determine recovery     * Fortified samples
  samples unfortified      of analyte at the        should be compared
  with the analyte of      various                  with the
  interest at a range      concentration.           unfortified sample
  of concentration.      * Recovery (%):            to assess the net
* Certified reference      ([C.sub.1]-              recovery of the
  materials (CRM).         [C.sub.2])/[C.sub.3]     fortification.
                           x 100                  * Recoveries from
                           Where,                   fortified samples
                           [C.sub.1] is             or matrix blanks
                           concentration            will usually be
                           determined in            better than real
                           fortified sample;        samples in which
                           [C.sub.2] is             the analyte is
                           concentration            more closely bound.
                           of unfortified         * Depending on how
                           sample; [C.sub.3] is     the CRM was
                           concentration of         produced and
                           fortification.           characterized, it
                                                    may be possible to
                                                    get >100% recovery.

References

(1.) A Laboratory Guide to Method Validation and Related Topics: The Eurachem Guide.

(2.) ISO 8402:1994 Quality--Vocabulary.

(3.) ASTM D 2777--86: Standard Practice for Determination of Precision and Bias of Applicable Methods of Committee D-19 on Water,

(4.) NABL 141: Guidelines for Estimation and Expression of Uncertainty in Measurement.

(5.) NABL 212: Guidance Document on Validation of Test Methods.

(6.) Eurachem Document on Quantifying Uncertainty in Analytical Measurement.