Evaluating the efficacy of managing West Coast groundfish resources through simulations.

Abstract--Management of West Coast groundfish resources by the Pacific Fishery Management Council involves Federal government and academic scientists conducting stock assessments, generally using the stock synthesis framework, applying the 40-10 rule to determine harvest guidelines for resources that are not overfished and conducting rebuilding analyses to determine harvest guidelines for resources that have been designated as overfished. However, this management system has not been evaluated in terms of its ability to satisfy the National Standard 1 goals of the Sustainable Fisheries Act. A Monte Carlo simulation framework is therefore outlined that can be used to make such evaluations. Based on simulations tailored to a situation similar to that of managing the widow rockfish (Sebastes entomelas) resource, it is shown that catches during recovery and thereafter are likely to be highly variable (up to [+ or -] 30% from one year to the next). Such variability is far greater than has been presented to the decision makers to date. Reductions in interannual variability in catches through additional data collection are, however, unlikely. Rather, improved performance will probably arise from better methods for predicting future recruitment. Rebuilding analyses include quantities such as the year to which the desired probability of recovery applies. The estimates of such quantities are, however, very poorly determined.

**********

National Standard 1 of the Sustainable Fisheries Act (SFA) of 1996 states that "Conservation and management measures shall prevent everfishing while achieving, on a continuing basis, the optimum yield from each fishery for the United States industry." The need to satisfy this National Standard has led inter alia to the requirement for the eight Regional Fishery Management Councils to develop control rules that are used to assess whether overfishing is occurring (1) or a stock is in an overfished state (e.g. Restrepo and Powers, 1999). In addition, the SFA specifies that a rebuilding plan has to be developed for any fish stocks that are designated as overfished. This plan needs to include the time period by which the stock will be rebuilt to [B.sub.MSY] (the average biomass associated with maximum sustainable yield, MSY), and the strategy by which the stock is to be rebuilt.

The Pacific Fishery Management Council (PFMC) has adopted the "4010" rule to manage groundfish stocks that are not designated as being overfished. This rule determines the harvest guideline for each groundfish stock by computing the catch corresponding to an [F.sub.MSY] proxy ([F.sub.40%.sup.2] for flatfish, [F.sub.50%] for rockfish in the Sebastes complex, and [F.sub.45%] for other species) and reducing it if the spawning output is estimated to be less than 40% of the estimated [B.sub.0]. This reduction in catch is linear with spawning output, being 0 at 0.4[B.sub.0] and 100% at 0.1[B.sub.0]. For stocks that are designated as being in an overfished state (defined for West Coast groundfish as being that the spawning output is less than 0.25[B.sub.0]) a rebuilding plan is developed. (3) The main features of the technical aspects of a rebuilding plan (referred to as a rebuilding analysis) identified by the Scientific and Statistical Committee of the PFMC are outlined in Appendix 1. In brief, the rebuilding analysis used by the PFMC involves projecting the best estimates of the current age-structure of the overfished population forward under a range of alternative fishing mortality rates and selecting the fishing mortality rate that has a Council-selected probability that the population recovers to the proxy for [B.sub.MSY] of 0.4[B.sub.0] within a time frame consistent with the specifications of the SFA.

Detailed stock assessments are available for only a small subset of the 81 species included in the PFMC Groundfish Management Plan. Of these species, nine (bocaccio [Sebastes paucispinis], canary rockfish [Sebastes pinniger], cowcod [Sebastes levis], darkblotched rockfish [Sebastes crameri], lingcod [Ophiodon elongates], Pacific ocean perch [Sebastes alutus], Pacific whiting [Merluccius productus], widow rockfish [Sebastes entomelas], and yelloweye rockfish [Sebastes ruberrimus]) have been designated overfished and rebuilding plans have been or are being developed for them. The direct consequences for industry of the implementation of a rebuilding plan can be substantial (e.g. a reduction in the catch of canary rockfish from 883 metric tons (t) in 1999 to only 90 t in 2001), although there are also indirect consequences in the form of reductions in the harvest of nonoverfished species to prevent overharvesting of overfished species through technical interactions.

The performance of the method commonly used for assessments of West Coast species has been evaluated to some extent (e.g. Sampson and Yin, 1998; Ianelli, 2002). However, the performance of this assessment method in combination with the rules used to determine harvest guidelines has not been evaluated.

Management procedures (4) are combinations of stock assessment methods and catch control laws that have been evaluated by means of Monte Carlo simulation to assess the extent to which they are able to satisfy the management objectives for a fishery. Evaluation of management procedures by means of Monte Carlo simulation has been argued to be essential because "if a management procedure is unable to perform adequately in the ideal world represented on a computer, what basis is there to assume that it will perform adequately in the real world?" (Sainsbury (5)). One caveat to this argument is that it is only possible to evaluate a management procedure if it is fully specified and if it will be followed for several years in reality.

Management procedures have been adopted by the International Whaling Commission for managing commercial and aboriginal whaling (e.g. IWC, 1992, 2001) and by southern African nations for managing a variety of pelagic and demersal resources (Butterworth and Bergh, 1993; Cochrane et al., 1998; Geromont et al., 1999). Management procedures are under consideration in Australia (Punt et al., 2001) and New Zealand (Starr et al., 1997). If it can be assumed that the same rules will be applied to modify rebuilding plans each time new information on abundance and year-class strength becomes available, it is possible to consider the combination of the assessment method, the default 40-10 rule, and rebuilding plans as a "management procedure" and evaluate it by means of Monte Carlo simulation. This study therefore involves determining from past practice the "management procedure" being applied by the PFMC. However, this "management procedure" has not been formally adopted in any way and the approach to managing West Coast groundfish could change in time.

This paper first outlines a simulation framework (a management procedure evaluation, MPE, framework) within which the expected performance of the approach used by the PFMC to determine harvest guidelines can be evaluated. It then evaluates variants of this approach for scenarios similar to that of managing the fishery fox widow rockfish.

Materials and methods

The steps in evaluating management procedures are as follows:

1 Identification of the management objectives and representation of these by using a set of quantitative performance statistics.

2 Identification of a range of alternative management procedures.

3 Development and parameterization of a set of alternative structural models (called operating models) of the system.

4 Simulation of the future use of each management procedure to manage the system (as represented by each operating model). For each year of the projection period, the simulations involve the following steps:

a Generation of the data available for assessment purposes.

b Application of a method of stock assessment to the generated data to determine key assessment-related quantities (e.g. current age-structure, spawning output in relation to target and limit levels, historical trends in recruitment) and any inputs to the catch control law.

c Application of the catch control law element of the management procedure to determine a harvest guideline.

d Determination of the biological implications of this harvest guideline by setting the catch for the "true" population represented in the operating model based on it. The step can potentially include "implementation uncertainty" (Rosenberg and Brault, 1993).

The harvest guideline is not updated every year in the simulations described in this article, but rather every third year (co-incident with the results from each new survey) and thus reflects the intended frequency with which assessments for West Coast groundfish species are conducted. Each simulation trial (i.e. each combination of an operating model variant and candidate management procedure) involves 100 simulations of an 80-year management period. The four steps listed above are discussed in detail below.

Note that for the application considered in this paper then, there are three "models": 1) the operating model that represents "reality" for the simulations, 2) an assessment model (a stock synthesis-like approach), and 3) a model to calculate the harvest guidelines. The data available to the last two models are generated from the first model.

The operating model

The operating mode] has been taken to be virtual]y identical to that on which the population assessments and rebuilding analysis calculations are based (Appendix 1), with two exceptions: 1) the approach used to generate recruitment and 2) the allowance for variability over time by the following double-logistic equation:

(1) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [S.sub.y,a] = the selectivity on fish of age a during year y;

[a.sup.1.sub.50], [a.sup.2.sub.50], [[delta].sub.1], [[delta].sub.2] = the parameters of the double-logistic equation;

[[gamma].sub.y] = the deviation from the average selectivity pattern in year y:

[[gamma].sub.y] = [[rho].sub.s][[gamma].sub.y-1]+[[epsilon].sup.s.sub.y] [[epsilon].sup.s.sub.y] ~ N(0;[[sigma].sup.2.sub.S],

[[rho].sub.S] = the interannual correlation in the deviation from average selectivity; and

[[sigma].sub.S] = a measure of the standard deviation of the interannual deviations from average selectivity.

Recruitment is assumed to be governed by a Beverton-Holt stock-recruitment relationship:

(2) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [R.sub.0] = the "virgin recruitment" (the number of zero-year-olds at the pre-exploitation equilibrium level);

[B.sub.y] = the spawning output at the start of year y;

h = the "steepness" of the stock-recruitment relationship (the fraction of virgin recruitment expected at 0.2[B.sub.0]); and

[[sigma].sub.R] = the standard deviation of the logarithms of the random fluctuations in recruitment about its expected value.

The biological parameters of the operating model are set to those for widow rockfish (Fig. 1A), and the catches for the 40 years prior to the year in which the management procedure is first applied (referred to as "projection year 1") are set to the actual catches for widow rockfish (Fig. 1B). The baseline values for the parameters h, [[sigma].sub.R], [[rho].sub.S], and [[sigma].sub.S] (Table 1) are educated guesses. The baseline choice for steepness, h, is lower than the posterior mean for this quantity (0.65) obtained by Dorn (2002) because, increasingly, West Coast rockfish are being found to be less productive than initially anticipated (e.g. Ianelli, 2002). The value assumed for the extent of variation in recruitment, [[sigma].sub.R], although based on the collection of estimates of this parameter by Beddington and Cooke (1983), is nevertheless also largely an educated guess. Sensitivity to the values for both h and [[sigma].sub.R] is explored.

[FIGURE 1 OMITTED]

The biomass at the start of year 1 is assumed equal to [B.sub.0], which is defined as the mean of the distribution for the unfished biomass which would arise given variability in recruitment about its expected value. However, this specification has little impact on the results. For example, the alternative that is defined to be the median of the distribution for the unfished biomass would only change [B.sub.0] by about 5%.

The value for [B.sub.0] for each simulation is selected so that the spawning output at start of year 41 (projection year 1) equals a prespecified fraction of [B.sub.0] (baseline fraction 0.2--i.e. just below the level that defines an overfished stock). Sensitivity to alternative values for the ratio of the spawning output at the start of year 41 to [B.sub.0] is explored (Table 1).

Generating future data

The data available for assessment purposes are survey indices of relative abundance, age-composition data from surveys, catch-rate-based indices of relative abundance, and age-composition data from the commercial catches. Table 2 lists the baseline specifications regarding the frequency at which the various data sources are collected and the parameters that determine the sampling variability associated with each data source.

The survey and catch-rate indices are generated by using the equations

(3a) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

(3b) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [B.sup.s,obs.sub.y] = the survey index for year y;

[B.sup.s.sub.a] = the survey selected-biomass during year y:

(4a) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[w.sub.a] = the mass of an animal of age a;

[S.sup.s.sub.a] = the selectivity of the survey gear on animals of age a (assumed to be governed by a logistic function and to be independent of time);

[N.sub.y,a] = the number of animals of age a at the start of year y;

[Z.sub.y,a] = the total mortality on animals of age a during year y;

[[sigma].sub.s] = the standard deviation of the random fluctuations in survey catchability;

[a.sub.max] = the oldest age considered in the operating model;

[I.sub.y] = the catch-rate index for year y;

[B.sup.e.sub.y] = is the exploitable biomass during year y;

(4b) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[F.sub.y] = the fully selected fishing mortality during year y; and

[[sigma].sup.c] = the standard deviation of the random fluctuations in fishery catchability.

Note that Equations 3a and 3b assume that the survey and fishery catchability coefficients are unity. This assumption can be made without loss of generality because the stock assessment method is not provided with this information and instead estimates these catchability coefficients. Note also that the key difference between the survey index and the catch-rate index is that selectivity for the latter changes over time (see Eq. 1), whereas selectivity for the former is time-invariant.

The age-composition data are generated by selecting a sample multinomially from the age-composition of the survey catch and of the fishery catch (see Eqs. 5a and 5b for the relative survey and fishery catches-at-age):

(5a) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

(5b) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

The PFMC management procedure

The "PFMC management procedure" (see Fig. 2 for an overview) involves first conducting a stock assessment by fitting an age-structured population dynamics model to the available data by maximizing a likelihood function. This approach mimics the common use of the stock synthesis framework (Methot, 2000) when conducting assessments of West Coast groundfish resources. The likelihood function is determined by assuming that the age-composition data are multinomially distributed (in the simulations with effective sample sizes given by the actual effective sample sizes) and by assuming that the survey and catch-rate series are log-normally distributed about the appropriate model quantities. The estimable parameters of the model are the annual recruitments, the annual fishing mortalities, the catchability coefficients, and the parameters that determine selectivity (the survey and fishery selectivity are [correctly] assumed to be governed by logistic and double-logistic equations). The values for the remaining parameters (weight-at-age, fecundity-at-age, and natural mortality) are assumed to be known without error. The key outputs from the assessment are time-series of recruitments and spawning out puts, and the age structure at the start of the last year of the assessment.

[FIGURE 2 OMITTED]

An estimate of the pre-exploitation equilibrium spawning output (i.e.[B.sub.0]) is obtained by multiplying the average recruitment for the first ten years of the assessment period by the spawning output-per-recruit in the absence of fishing. This approach to estimating [B.sub.0] has been used for several rebuilding analyses for West Coast groundfish species. If the estimate of the current spawning output exceeds 0.4[B.sub.0] or if it exceeds 0.25 [B.sub.0] and the resource is not currently under rebuilding (i.e. has not yet been declared to be in an overfished state), a raw harvest guideline is computed using the 40-10 rule. On the other hand, if the estimate of the current spawning output is less than 0.25 [B.sub.0] or the stock is currently under a rebuilding plan and the spawning output has not yet recovered to 0.4 [B.sub.0], the raw harvest guideline is based on the application of the rebuilding analysis (see Appendix 1 for further details).

It is necessary to know the maximum possible rebuilding period, [T.sub.max], when using a rebuilding analysis to calculate a harvest guideline. If the stock is declared overfished in the present year, [T.sub.max] is computed as described in Appendix 1. On the other hand, if the stock is currently under a rebuilding plan, [T.sub.max] is taken to be the value computed when the stock was first declared overfished. Therefore, the implementations of the rebuilding plans considered in this paper are based on the assumption that the [T.sub.max] and the probability of recovery by [T.sub.max] are set when the first rebuilding analysis is conducted and not changed thereafter. The probability of recovery by [T.sub.max] is taken to be 0.6 in this paper because this is the probability on which management of widow rockfish is currently based. This probability ranges between 0.55 and 0.92 among the seven overfished groundfish resources for which it has been selected by the PFMC.

Calculation of a harvest guideline using the 40-10 rule and application of the rebuilding analysis requires the ability to generate future recruitment. For the purposes of the present study (and consistent with current practice), future recruitment is either generated from the estimates of recruitment from the assessment or by multiplying the spawning output by a generated value for the recruits-per-spawning output ratio. The pool of recruitment to recruits-per-spawning output is taken to be those for the last 23 years of the assessment period less those for the last three years. The last three years are excluded because of their known poor precision. The approach used to generate recruitment therefore leads to the set of recruitments used to conduct projections changing with time. Allowing the set of recruitments to change with time is needed to avoid an inconsistency between the recruitments used for projections and the recruitments on which the estimate of [B.sub.0] is based.

Allowance is made for the raw harvest guideline to be constrained so as not to change by more than a prespecified percentage from that for the previous year and not to fall outside of specified limits, although this option is not part of the baseline simulations.

One aspect of the actual management process that is ignored in the simulation of the PFMC management procedure is the time-lag between the collection of data and their use in assessments (for example, catch-at-age information from surveys conducted in one year would usually not be available for use in the assessments conducted in the following year) and that between assessments being conducted and their being used for management purposes.

The performance statistics

A variety of performance statistics are considered (Table 3). These consider both the performance of the management procedure in terms of the behavior of the rules used for management (statistics [F.sub.rec], [Y.sub.rec], and [P.sub.decl]) and of satisfying the goals established by the SFA in relation to the status of the population and the fishery (statistics 5%D, 50%D, [bar]C, AAV, and [P.sub.rec). The choice of years 20 and 60 in the definitions of the latter five statistics is meant to capture "short"-term and medium-term considerations. For instance, recovery should have occurred by year 60 in most cases and the population should be well above 0.25[B.sub.0] after 20 years. The catch and catch variability statistics for the first 20 years provide an indication of the likely impacts of recovery on the industry.

The need to examine aspects of the behavior of the management rules can be understood from Figure 3, which shows results for four simulations for the combination of a PFMC management procedure and an operating model variant. The solid lines are the "true" time-trajectories of spawning output (expressed in relation to the pre-exploitation level) and the dotted lines reflect the estimates of this ratio each time an assessment is conducted (every third year for the analyses shown in Fig. 3). The up arrows indicate when the assessment first indicates that the population is overfished (based on the model estimates of spawning output)--note that a population may be identified. To be overfished more than once during a given simulation. The down arrows indicate the years in which recovery is predicted by the rebuilding analysis software (with the estimates from the assessment) to occur with 60% probability. The solid bar parallel to the x-axis indicates the years in which management is based on the rebuilding plan (rather than the 40-10 rule). The bar will stretch from the up arrow to the down arrow unless the population is assessed to have recovered to 0.4[B.sub.0] (when management reverts to being based on the 40-10 rule).

[FIGURE 3 OMITTED]

There are several possible impacts of the difference between the perceived and true state of the system. For example, the population can erroneously be assessed not to be overfished in the first projection year (e.g. simulation 1 in Fig. 3). The statistic [P.sub.decl] is designed to capture the frequency of this possibility. Even if the population is assessed to be overfished, there is no guarantee that it will recover with the expected probability and in the "correct" year. For example, for simulation 1, the stock assessment indicates that recovery occurs in year 71 (the solid bar consequently stops in year 71) even though the true population size is less than 30% of [B.sub.0] at that time. The statistic [F.sub.rec] attempts to capture whether the rebuilding analysis performs as expected given that the population is assessed to be overfished at the start of the first projection year.

There are other aspects to evaluating the behavior of the management rules in relation to the perceived and true state of the system (e.g. the difference between the true and estimated biomasses and recruitments). Although it is straightforward to evaluate these aspects (e.g. Patterson and Kirkwood, 1995; Punt et al., 2002), they are not considered in detail in this paper to reduce the volume of results presented.

Results and discussion

Detailed results for a single operating model variant and management procedure

Figures 4 and 5 and Table 4 summarize aspects of a simulation trial in which the operating model has its baseline parameterization (Tables 1 and 2) and in which the management procedure used to set harvest guidelines is the PFMC management procedure with no constraints on interannual variation in harvest guidelines other than an upper limit of 10,000 t. The lack of any constraints on changes in harvest guidelines has been imposed because the PFMC has not adopted any such constraints. The harvest guideline is updated every third year.

[FIGURES 4-5 OMITTED]

Figure 4 shows the time-trajectories of catch, spawning output in relation to the pre-exploitation equilibrium level ("true" and estimated), and the perceived fishing mortality on which the harvest guideline is based for three of the 100 simulations that constitute a simulation trial. The horizontal bars on the x-axis again reflect the year during which the stock is managed by using the results from the rebuilding analysis rather than the 40-10 rule. The most notable feature of Figure 4 is the high variability in annual catches. This variability arises for several reasons: 1) the additional information on population biomass obtained each time a survey occurs changes the perceived status of the resource and hence how far the spawning output is from the target level of 0.4[B.sub.0]; 2) an extension of the assessment period changes the set of recruitments on which generation of future recruitment is based; and 3) a change from being under a rebuilding plan to being managed by means of the 40-10 rule can lead to marked changes in catch. The latter is evident by the change in fishing mortality and catch when the spawning output is estimated to reach 0.4[B.sub.0] (i.e. the end of the horizontal bar). A marked impact due to the addition of data for a single 3-year period may appear surprising. However, effects of this nature have already been observed for West Coast species (see, for example, the 2002 update to the sablefish [Anoplopoma fimbria] stock assessment [Schirripa and Methot (6)]).

The extent of variability in catch in Figure 4 differs markedly from the way advice on expected catches during the rebuilding period is presented to the decision makers (e.g. Fig. 6). One way to improve the presentation of in formation on expected catches would be to include some individual catch trajectories from those on which the rebuilding analysis is based. However, even these would severely underestimate the actual extent of uncertainty because they are conditioned on knowing the age-structure of the population at the start of the projection period and are based on fixed levels of fishing mortality during the rebuilding period.

[FIGURE 6 OMITTED]

The impact of estimation uncertainty is also evident in Figure 4. The following are three examples of this: 1) management based on the rebuilding plan only starts in year 53 in simulation 1 because, prior to this year, the stock assessment indicates (erroneously) that the stock is above rather than below 0.25[B.sub.0]; 2) the resource is predicted to have recovered to 0.4[B.sub.0] in year 71 in simulation 1 (and hence management is based on the 40-10 rule thereafter)--however, the spawning output is really only slightly larger than 0.3[B.sub.0] at this time; and 3) in simulation 3 the assessment model indicates that the spawning output has recovered to above 0.4[B.sub.0] in year 65 when, in fact, it recovered to 0.4[B.sub.0] three years earlier.

The results of all 100 simulations are summarized by the time-trajectories in Figure 5. The trajectories of catch in Figure 5 are notably less variable that the individual trajectories in Figure 4 because, for instance, the 5th, median, and 95th intervals for the catch in year 80 are obtained by sorting all 100 year-80 catches and taking the appropriate percentiles. Unlike the individual trajectories, the median trajectories of catch and spawning output show quite smooth changes over time. This result highlights the importance of the AAV statistic that captures interannual variation in catches within individual simulations.

Overall, there is a high probability (0.82) that the assessment model identifies that the spawning output is less than 0.25[B.sub.0] at the start of the projection period (Table 4). However, the probability that recovery occurs at or before the [T.sub.max] year predicted from the rebuilding analysis conducted in projection year 1 is rather low (0.22) and 50% of simulations exceed 0.4[B.sub.0] only in year 72 (i.e. after 30 years). The probability of being below the overfished level of 0.25[B.sub.0] still exceeds 5% after 60 years of management with this management procedure although there is an 80% probability that the spawning output recovers to 0.4[B.sub.0] sometime during the first 60 years of management with the management procedure.

It should be noted that the impact of recruitment variability and assessment errors following recovery to 0.4[B.sub.0] can be consequential. For example, the probability of having reached 0.4[B.sub.0] after 60 years of management by using the management procedure exceeds 0.8 but the median value of the ratio of the spawning output in year 60 to [B.sub.0] is nevertheless still less than 0.4 (Table 4, Fig. 5). One reason for the spawning output not stabilizing at 0.4 [B.sub.0] is a discrepancy between the fishing mortality rate that stabilizes the population at [B.sub.0] (deterministically) and [F.sub.50%]. For the baseline steepness of 0.4, the fishing mortality required to stabilize the spawning output at 0.4 [B.sub.0] actually corresponds to a lower fishing mortality than [F.sub.50%] (closer to [F.sub.63%]).

Sensitivity to alternative management procedures

Table 4 includes results for a range of variants of the baseline management procedure designed to improve its performance. The following are areas where improved performance is desirable: 1) the extent of interannual variability in catches; 2) the similarity between the year in which the rebuilding analysis indicates recovery will occur and the year at or before which it actually occurs; and 3) the probability of being below the overfished level after 20 and 60 years.

The first variant of the baseline management procedure ("with constraints" in Table 4) involves imposing maximum and minimum catch limits of 30 and 8000 t and constraining changes in harvest guideline not to exceed 25% from one year to the next, except in the first year when reductions of up to 99% are allowed. This variant leads to much lower interannual variation in catches when a 60-year period is considered (17% compared with 25%) but the AAV is actually higher for the first 20 years. This variant also leads to higher probabilities of recovery. However, there is still a large discrepancy between the actual year of recovery to 0.4[B.sub.0] and the year that underlies the management procedure (the value of [F.sub.rec] in Table 4 is only 0.27 for the "with constraints" variant).

The second variant considered ("no 10 year and estimated [F.sub.MSY]" see Table 4) drops the requirement that [T.sub.max] be defined as 10 years if the resource can be recovered in 10 years and instead always sets [T.sub.max] to [T.sub.min] plus one mean generation. It also allows the [F.sub.MSY] proxy used when applying the 40-10 rule to differ from the default value of [F.sub.50%] by setting it to [F.sub.rep] (Jakobsen, 1993) if [F.sub.rep] is lower than [F.sub.MSY]. Estimating (rather than fixing) [F.sub.MSY] is consistent with the recommendation of Brodziak (2002). The major performance difference between this variant and the baseline management procedure is the increased value of [F.sub.rec].

The "preferred" variant in Table 4 combines the features of the "with constraints" and "no 10 years and estimated [F.sub.MSY]" variants. Compared with the baseline management procedure, it leads to a markedly increased value for [F.sub.rec] (remarkably close, in fact, to the desired value of 0.6), slightly lower catch variability, a less than 5% chance of being overfished after 20 years, and higher probabilities of being recovered to 0.4[B.sub.0] after 20 and 60 years of management. The major disadvantage of this variant is the lower catches and that it leaves the spawning output well above 40% of [B.sub.0] after 60 years (see row "preferred" in Table 4).

Prior to the adoption of Amendment 11 of its Groundfish Management Plan, the PFMC set harvest guidelines using only the 40-10 rule. (7) Table 4 therefore also lists results for management procedures based on the 40-10 rule. When the 40-10 rule is applied without any constraints ("PFMC (baseline)" in Table 4), the probability of recovery and the values for the "50%D" statistic are lower (particularly the former) than for the "preferred" variant. In contrast, application of the 40-10 rule with constraints ("PFMC (preferred)" in Table 4) leads, arguably, to no more than a slight difference in catch (the 40-10 rule achieves higher catches) and probability of recovery (the "preferred" variant achieves a higher probability of recovery). The remaining analyses of this paper focus on the "preferred" variant. Future consideration of management procedures for West Coast groundfish resources should consider a management procedure that is based simply on the 40-10 rule and has no associated rebuilding analysis component, at least for comparative purposes. At present, however, such a management procedure would be inconsistent with the SFA because it would not specify the time to recover to the proxy for [B.sub.MSY] (even if the results of this paper suggest that there is considerable uncertainty associated with the estimation of this particular quantity).

Sensitivity to alternative operating model specifications

The values assumed for h and [[sigma].sub.R] in the baseline operating model are somewhat arbitrary. Table 5 therefore examines the sensitivity of the results for the "preferred" management procedure to varying the values assumed for these parameters, as well as that of the size of spawning output at the start of the first projection year to [B.sub.0].

The results are, as expected, sensitive to all three of the factors considered. Increasing [[sigma].sub.R] from 0.4 through 0.6 to 1 leads to lower and more variable catches, a slightly higher probability of recovery in the first 20 years and a markedly higher value of 50%D after 60 years (0.74 for [[sigma].sub.R]=1 compared to 0.46 for [[sigma].sub.R]=0.4). The ability to detect an overfished stock declines slightly as the extent of variation in recruitment increases. The management procedure behaves as expected as steepness is increased from 0.25 through 0.4 to 0.7; the probability of recovery is markedly higher for high values of steepness even though the management procedure does identify cases with low steepness, and accordingly sets very low harvest guidelines in such cases. However, it is perhaps noteworthy that the probability of correctly identifying that the resource is overfished is lowest for the least productive scenario. The catches for the scenario in which the spawning output is 10% of [B.sub.0] at the start of the first projection year are much lower than for the baseline scenario, particularly over the first 20 years. However, these lower catches are necessary to achieve recovery (the median value of the statistic 50%D after 60 years is 0.52 and there is a 0.93 probability of the spawning output having recovered to 0.4[B.sub.0] after 60 years for this scenario).

The behavior of the management procedure can be evaluated in terms of whether it eventually allows the stock to recover to 0.4[B.sub.0] and whether it keeps the stock away from the overfished level of 0.25[B.sub.0]. The "preferred" management procedure can be argued to satisfy this criterion, except possibly for the scenario with the lowest steepness but, even in this case, the probability of recovery is 0.6 after 60 years.

The value for the [F.sub.rec] statistic varies markedly depending on steepness and the ratio of the spawning output at the start of the first projection year to [B.sub.0]. Although the "preferred" management procedure performs well for the baseline scenario in terms of recovering the resource by the predicted value for [T.sub.max], this good performance is clearly a fortunate anomaly. However, it does help to highlight that predictions of the year-to-recovery from rebuilding analyses should be interpreted with considerable caution.

Sensitivity to data quality

The data-related specifications for the baseline trial (Table 2) could be considered to be data-rich. It is therefore important to assess the sensitivity of the results to the quality of the data. The row "deterministic data" in Table 5 provides results for a trial in which the survey biomass index, the catch-rate index, and the age-composition data are known without error. The results from this trial provide an upper bound on the impact of improved data quality on the assessment results. (8) Somewhat surprisingly, the results for this trial are not notably better than for the baseline trial--the most notable difference between the baseline trial and the "deterministic data" trial being the higher values for the "5%D" statistics for the latter trial. The lack of major improvement in performance arises because, even with perfect information on spawning output and recruitment, it is still not possible to estimate [B.sub.0] exactly by multiplying average recruitment for the first 10 years of the assessment period by spawning output-per-recruit in the absence of fishing (hence the value of 0.84 for [P.sub.decl])" Furthermore, the rebuilding analyses are still based on generating future recruitment by using spawning output and recruitment data for only 20 years, which is clearly a major source of variability in the predictions from the rebuilding analysis.

Decreasing the catch-at-age sample size from 200 to 50 has relatively little impact on the values for the performance statistics (the AAV statistic is marginally higher and the average catch, particularly for the 20-year projection horizon, is lower). Decreasing the precision of the catch-rate data has a rather larger impact. This is most evident in the value for the "5%D" statistic which is 0.2 rather than 0.25, as is the case for the baseline trial. The "5-yr update frequency" scenario in Table 5 examines the implications conducting assessments every fifth rather than every third year. The results are not markedly sensitive to the interassessment period although the lower values for the "5%D" statistics are perhaps noteworthy.

General remarks

The framework developed in this paper provides an objective basis for contrasting different management procedures and evaluating their sensitivity to uncertainty. Given such a framework, it becomes possible to compare variants of one class of management procedure (e.g. Table 4) and to compare variants among different classes of management procedure.

The management procedure options presented in this paper are but a small subset of those possible. In particular, it should be possible to improve performance by modifying the approach used to generate future recruitment when conducting rebuilding analyses to make use of some form of stock-recruitment relationship. One reason for expected improved performance is that it may then be feasible to estimate the fishing mortality rate corresponding to 0.4[B.sub.0] rather than having to set it to the default value of [F.sub.50%] or basing it on [F.sub.rep]. Other possible management procedure options include 1) not increasing the rebuilding fishing mortality rate selected when the rebuilding analysis was first conducted if a stock is recovering faster than initially anticipated; 2) not decreasing the rebuilding fishing mortality rate as long of the probability of recovery by [T.sub.max] is at least 0.5; and 3) smoothing the discontinuity that arises when a stock changes status from being under a rebuilding plan to being managed with the 40-10 rule when the stock has recovered to 0.4[B.sub.0]. In terms of the last option, one of the issues considered an early rebuilding analysis for widow rockfish involved fishing mortality increasing to its target level as the stock approaches 0.4[B.sub.0] (MacCall (9)).

The values for the [F.sub.rec] statistic highlight that the predictions of the time to recovery (even in a probabilistic sense) from rebuilding analyses are highly uncertain. The uncertainty of this estimate of the time to recovery is due to the uncertainty about current stock size and that associated with making long-term predictions based on a short time-series of spawning output and recruitment data.

Although the performance of the management procedures is less than ideal, the results are almost certainly optimistic because the operating model is extremely simple and considers no major structural uncertainties (except for variability in selectivity over time). In contrast, Punt et al. (2002) found that including spatial structure in an operating model and assessing the stock by using a spatially aggregated assessment approach led to assessments that were markedly in error. However, the simulations conducted by Punt et al. (2002) were developed for a far more data-poor situation than that for West Coast groundfish, although there is also clearly spatial structure in the West Coast groundfish fishery. Another source of uncertainty not considered in this paper but that may be of critical importance to the management of West Coast groundfish species is the impact of environmental regime shifts, which have been argued to impact long-term trends in recruitment (e.g. Francis et al., 1998).

An important aspect of this study is the ability to focus on the relationship between the overall performance of a management procedure and the performance of its constituent parts. For example, the results for the "deterministic data" scenario in Table 4 show that given the approach used to conduct the future projections, even perfect information from surveys and very large age-composition samples are unlikely to lead to marked improvements over the current situation if that situation is adequately modeled by the baseline operating model. Identification of the key sources of uncertainty could be used to focus future management-related research activities.

The computational requirements of the calculations outlined above are substantial. In particular, the need to apply a fairly complicated method of stock assessment once every three years means that rapid evaluation of management procedures is (currently) computationally not feasible. It is possible, in principle, to simplify the management procedure considerably by assuming that the results from a stock assessment can be mimicked by generating a biomass estimate based on the "true" biomass but with some random error (e.g. Hilborn et al., 2002). However, although such an approach may be satisfactory for some management procedures (e.g. those that set the harvest guideline equal to some fraction of the current biomass), this is not the case for PFMC-type management procedures that depend on the (assessed) age-structure of the population.

It needs to be recognized that any simulation study is by design case-specific. However, the conclusions of this study may be relevant to a fairly broad set of West Coast rockfish species owing to their similar biology and exploitation history--the two factors most likely to impact the relative performance of different management procedures.

Appendix 1 : An overview of the technical aspects of the PFMC's rebuilding analysis

The key steps of the PFMC's rebuilding analysis are 1) to select the maximum allowable rebuilding time ([T.sub.max]) 2) to develop specifications for projecting the population size at the start of the current year-to-year [T.sub.max], and 3) to calculate the target fishing mortality rate so that the probability of the spawning output rebuilding to 0.4[B.sub.0] at or before [T.sub.max] equals a prespecified value, [P.sub.rec] (taken to be 0.6 for purposes of the present study).

Projecting the population forward and defining [B.sub.0]

The population projections are based on the equation

(A.1) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [N.sub.y,a] = the number of animals of age a at the start of year y;

M = the instantaneous rate of natural mortality (assumed to be independent of age);

[S.sub.a] = the selectivity for animals of age a;

F = the fully selected (i.e. [S.sub.a] [right arrow] 1) fishing mortality;

[R.sub.y] = the recruitment (both sexes) during year y;

[a.sub.min] = the lowest age class considered in the model; and

[a.sub.max] = the oldest age class considered in the model (treated as a plus-group).

The age structure of the population at the start of the first year of the projection period is taken to be that from the most recent assessment. A variety of approaches are available to generate future recruitment (PFMC (10)). However, for consistency with the approach used in the bulk of the rebuilding analyses conducted to date, future recruitment is either based on randomly sampling recruitments (with replacement) from a prespecified historical period or based on randomly sampling the ratio of the recruitment to the spawning output that spawned that recruitment (with replacement) and then multiplying by current spawning output. The choice between basing the projections on sampling recruitments or sampling recruits-per-spawning output is determined by regressing each of these on time and selecting whichever has the lesser slope. The reason for doing this is that the lack of a trend in recruits-per-spawning output is indicative of a stock-recruitment relationship with low "steepness" (Francis, 1992), whereas the lack of a trend in recruitment is indicative of a stock-recruitment relationship with high "steepness."

The pre-exploitation equilibrium spawning output used to determine the rebuilding target is computed by multiplying the unfished spawning output-per-recruit by the average recruitment over a prespecified number of historical years. Note that the range of years on which to base the estimate of [B.sub.0] will usually differ from that on which generation of future recruitment is based.

It should also be noted that no account is taken of uncertainty regarding the current age structure, natural mortality, selectivity, etc., although the projections do account for uncertainty about future recruitment

Selecting the maximum allowable rebuilding period

The maximum allowable rebuilding time, [T.sub.max], is defined as the maximum of 10 years and the sum of the mean generation time and the minimum possible rebuilding time. This specification implements the requirement of the SFA to "take into account the status and biology of any overfished stocks offish, [and] the needs of fishing communities." The minimum possible rebuilding period for a giver future projection is computed by projecting the population forward with zero fishing mortality and by identifying the year in which the spawning output first reaches 0.4[B.sub.0]. [T.sub.min] is the median of the distribution for this year constructed by conducting projections for many different (random) realizations of future recruitment.

Calculating the target fishing mortality rate

The target fishing mortality rate and hence the harvest guideline are determined by projecting the population forwards many times (100 times for the purposes of this paper), each time with a different sequence of future recruitment and for a variety of alternative Fs and then identifying the level of F that corresponds to the spawning output having reached 0.4[B.sub.0] by [T.sub.max] with the prespecified probability [P.sub.rec']

Table 1

The baseline parameters of the operating model and the values
used in the tests of sensitivity. N/A = not available.

Parameter          Baseline value    Sensitivity values

[[rho].sub.S]         0.707             N/A
[[sigma].sub.S]       0.4               N/A
h                     0.4               0.25; 0.7
[[sigma].sub.R]       0.6               0.4; 1
M                     0.15/yr           N/A
Spawning output       0.2[B.sub.0]      O.1[B.sub.0]; 0.4[B.sub.0]
  in year 41

Table 2

The parameters on which the generation of future data is based.
[n.sup.e] is the sample size for the multinomial distribution.

                 First year
Data source      collected    Frequency          Precision

Catch rates          14       Every year         [[sigma].sup.c] = 0.4
Fishery age-
  composition        21       Every year         [n.sup.e] = 200
Survey indices       13       Every third year   [[sigma].sub.s] = 0.5
Survey age-
  composition        13       Every third year   [n.sup.e] = 200

Table 3

The performance statistics used in the present study. For consistency
with the definition of recovery used by the Pacific Fishery Management
Council, "recovery by year x" is defined as the spawning output being
larger than 0.4[B.sub.0] at or before year x. Some of the statistics
are based on the "actual" (i.e. operating model) spawning output and
others are based on the "assessed" (i.e. assessment model) spawning
output.

Abbreviation                      Description

[F.sub.rec]     The fraction of the simulations in which the stock is
                assessed to be overfished at the start of the first
                projection year that actually recover by the maximum
                possible recovery year determined from the
                rebuilding analysis conducted in projection year 1.
[Y.sub.rec]     The median year in which the actual spawning output
                first reaches 0.4[B.sub.0].
[P.sub.decl]    The proportion of simulations in which the spawning
                output is assessed to be below 0.25[B.sub.0] (i.e.
                overfished) at the start of projection year 1.
5%D/50%D        The lower 5th and median of the distribution of the
                actual spawning output in projection years 20 and 60
                expressed in relation to the actual pre-exploitation
                spawning output, [B.sub.0].
AAV             Average annual absolute change in catch evaluated
                after 20 and 60 years, i.e.
                AAV = [summation over (y)] [absolute value of
                [C.sub.y] - [C.sub.y-1]]/[summation over y] [C.sub.y],
                where [C.sub.y] is the catch during year y.
[bar]C          Average annual catch over projection years 1-20 and
                1-60.
[P.sub.rec]     The fraction of simulations in which actual spawning
                output reached 0.4[B.sub.0] sometime between
                projections years 1 and 20 and between projection years
                1 and 60 (but may have dropped below 0.4[B.sub.0]
                again).

Table 4

Performance statistics (see Table 3 for definitions) for six
alternative management procedure variants. All of the calculations
in this table relate to the baseline operating model. PFMC =
Pacific Fishery Management Council. N/A = not applicable.

Management procedure        [F.sub.rec]   [Y.sub.rec]   [P.sub.decl]

Baseline                       0.22           72            0.82
With constraints               0.27           61            0.82
No 10 years and estimated
  [F.sub.MSY]                  0.42           68            0.82
Preferred                      0.59           62            0.82
PFMC (baseline)                N/A            95            N/A
PFMC (preferred)               N/A            64            N/A

                                     Results after 20 years

Management procedure        5%D    50%D   AAV    [bar]C   [P.sub.rec]

Baseline                    0.22   0.33   0.33    1759       0.32
With constraints            0.24   0.40   0.38     591       0.54
No 10 years and estimated
  [F.sub.MSY]               0.24   0.34   0.30    1652       0.27
Preferred                   0.25   0.39   0.31     950       0.49
PFMC (baseline)             0.19   0.29   0.23    2273       0.07
PFMC (preferred)            0.23   0.36   0.30    1239       0.45

                                     Results after 60 years

Management procedure        5%D    50%D   AAV    [bar]C   [P.sub.rec]

Baseline                    0.23   0.36   0.25    2847       0.80
With constraints            0.24   0.41   0.17    2440       0.89
No 10 years and estimated
  [F.sub.MSY]               0.25   0.41   0.24    2649       0.84
Preferred                   0.28   0.54   0.21    1961       0.96
PFMC (baseline)             0.24   0.33   0.20    2851       0.55
PFMC (preferred)            0.30   0.48   0.20    2265       0.93

Table 5

Performance statistics (see Table 3 for definitions) for 10 variants of
the baseline operating model. All of the calculations in this table
relate to the preferred management procedure. N/A = not applicable.

Operating model scenario       [F.sub.rec]   [Y.sub.rec]   [P.sub.decl]

Baseline                          0.59           62            0.82
Structural changes
  [[sigma].sub.R]=0.4             0.59           63            0.86
  [[sigma].sub.R]=1               0.59           61            0.72
  h=0.25                          0.15           94            0.76
  h=0.7                           0.84           53            0.87
  Initial spawning out = 0.1
    [B.sub.0]                     0.42           72            1.00
  Initial spawning out = 0.4
    [B.sub.0]                     N/A           N/A            N/A
Data-related changes
  Deterministic data              0.68           61            0.84
  [n.sup.e]=50                    0.68           60            0.82
  [[sigma].sup.c]=1               0.56           62            0.79
  5-yr update frequency           0.55           62            0.80

                                      Results after 20 years

Operating model scenario           5%D          50%D           AAV

Baseline                          0.25          0.39           0.31
Structural changes
  [[sigma].sub.R]=0.4             0.24          0.38           0.26
  [[sigma].sub.R]=1               0.23          0.41           0.43
  h=0.25                          0.20          0.28           0.76
  h=0.7                           0.31          0.46           0.16
  Initial spawning out = 0.1
    [B.sub.0]                     0.19          0.29           0.43
  Initial spawning out = 0.4
    [B.sub.0]                     0.31          0.50           0.21
Data-related changes
  Deterministic data              0.29          0.38           0.30
  [n.sup.e]=50                    0.26          0.39           0.32
  [[sigma].sup.c]=1               0.20          0.39           0.31
  5-yr update frequency           0.21          0.38           0.27

                                     Results after        Results after
                                       20 years             60 years

Operating model scenario         [bar]C      [P.sub.rec]       5%D

Baseline                           950          0.49           0.28
Structural changes
  [[sigma].sub.R]=0.4             1242          0.44           0.25
  [[sigma].sub.R]=1                417          0.54           0.32
  h=0.25                            86          0.02           0.23
  h=0.7                           3427          0.93           0.40
  Initial spawning out = 0.1
    [B.sub.0]                      417          0.05           0.27
  Initial spawning out = 0.4
    [B.sub.0]                     2881          0.92           0.30
Data-related changes
  Deterministic data               957          0.51           0.31
  [n.sup.e]=50                     785          0.56           0.29
  [[sigma].sup.c]=1                987          0.48           0.31
  5-yr update frequency           1160          0.49           0.27

                                       Results after 60 years

Operating model scenario          50%D           AAV          [bar]C

Baseline                          0.54          0.21           1961
Structural changes
  [[sigma].sub.R]=0.4             0.46          0.18           2379
  [[sigma].sub.R]=1               0.74          0.32            592
  h=0.25                          0.38          0.50            126
  h=0.7                           0.61          0.14           3951
  Initial spawning out = 0.1
    [B.sub.0]                     0.52          0.23           1375
  Initial spawning out = 0.4
    [B.sub.0]                     0.66          0.19           2849
Data-related changes
  Deterministic data              0.55          0.20           2050
  [n.sup.e]=50                    0.55          0.22           1938
  [[sigma].sup.c]=1               0.57          0.22           1962
  5-yr update frequency           0.53          0.19           1980

                               Results after
                                 60 years

Operating model scenario       [P.sub.rec]

Baseline                          0.96
Structural changes
  [[sigma].sub.R]=0.4             0.87
  [[sigma].sub.R]=1               0.96
  h=0.25                          0.60
  h=0.7                           1.00
  Initial spawning out = 0.1
    [B.sub.0]                     0.93
  Initial spawning out = 0.4
    [B.sub.0]                     0.97
Data-related changes
  Deterministic data              0.98
  [n.sup.e]=50                    0.97
  [[sigma].sup.c]=1               0.97
  5-yr update frequency           0.95

Acknowledgments

Discussions with Alec MacCall, John DeVore, and Richard Methot are gratefully acknowledged as are the comments on an earlier version of this paper by Pamela Mace and two anonymous reviewers. This work was funded through NMFS grant NA07FE0473.

(1) In the present study, and consistent with usage by the Pacific Fishery Management Council, "overfishing" means that the level of fishing mortality exceeds that associated with MSY and "being in an overfished state" means that the current spawning output is less than 25% of the pre-exploitation equilibrium spawning output, [B.sub.0] (spawning output is the product of egg production-at-age and numbers-at-age).

(2) [F.sub.x%] is the fishing mortality rate at which the spawning output-per-recruit is reduced to x% of its unfished level.

(3) One implication of this is that the 40-10 rule is not actually used if the stock is assessed to be below 0.25[B.sub.0].

(4) Also referred to as "harvest strategies" (Punt et al., 2001), "management decision rules" (Starr et al., 1997), "fisheries control systems" (Hilborn, 1979), and "operational management procedures" (Barnes, 1999).

(5) Sainsbury, K.G. 2001. Personal commun. CSIRO Marine Research, Castray Esplanade, Hobart, TAS 7000, Australia.

(6) Schirripa, M. J., and R. Methot. 2002. Status of the sablefish resource off the continental U.S. Pacific Coast in 2001. In Stock assessment and fishery evaluation: appendix to the status of the Pacific Coast groundfish fishery through 2001 and acceptable biological catches for 2002, x + 122 p. Pacific Fishery Management Council, 7700 NE Ambassador Place, Portland, OR 97220.

(7) Albeit with different target fishing mortality levels.

(8) The assessment still ignores interannual changes in selectivity; therefore the assessment results will not be exactly the same as the true values.

(9) MacCall, A.D. 2002. Personal commun. NMFS Santa Cruz Laboratory, 110 Shaffer Rd, Santa Cruz, CA 95060.

(10) PFMC (Pacific Fishery Management Council). 2001. SSC terms of reference for groundfish rebuilding analysis, 9 p. Pacific Fishery Management Council, 7700 NE Ambassador Place, Portland, OR 97220.

Literature cited

Barnes, W. R.

1999. Viewpoint: an industry view of the application of operational management procedures to setting total allowable catches for the South African pelagic fishery. ICES. J. Mar. Sci. 56:1067-1069.

Beddington, J. R., and J. G. Cooke.

1983. The potential yield of fish stocks. FAO Fish. Tech. Pap. 242:1-47.

Brodiak, J.

2002. In search of optimal harvest rates of west coast groundfish. N. Am. Fish. Manage. 22:258-271

Butterworth, D. S., and M. O. Bergh.

1993. The development of a management procedure for the South African anchovy resource. In Risk evaluation and biological reference points for fisheries management (S. J. Smith, J. J. Hunt, and D. Rivard, eds.), p. 83-89. Can. Spec. Publ. J. Fish. Aquat. Sci. 120.

Cochrane, K. L., D. S. Butterworth, J. A. A. De Oliveira, and B. A. Roel.

1998. Management procedures in a fishery based on highly variable stocks and with conflicting objectives: experiences in the South African pelagic fishery. Rev. Fish. Biol. Fish. 8:177-214.

Dorn, M. W.

2002. Advice on west coast rockfish harvest rates from Bayesian meta-analysis of stock-recruit relationships. N. Am. Fish. Manage. 22:280-300.

Francis, R. I. C. C.

1992. Use of risk analysis to assess fishery management strategies: a case study using orange roughy (Hoplostethus atlanticus) on the Chatham Rise, New Zealand. Can. J. Fish. Aquat. Sci. 49:922-930.

Francis, R. S., A. Hare, A. Hollowed, and W. Wooster.

1998. Effects of interdecadal variability on the oceanic ecosystems of the NE Pacific. Fish. Oceanogr. 7:1-21.

Geromont, H. F., J. A. A. De Oliveira, S. J. Johnston, and C. L. Cunningham.

1999. Development and application of management procedures for fisheries in southern Africa. ICES J. Mar. Sci. 56:953-966.

Hilborn, R.

1979. Comparison of fisheries control systems that utilize catch and effort data. J. Fish. Res. Board Can. 36: 1477-1489.

Hilborn, R., A. Parma, and M. Maunder.

2002. Exploitation rate reference points for west coast rock fish: are they robust and are there better alternatives? N. Am. Fish. Manage. 22:365-375.

Ianelli, J. N.

2002. Simulation analyses testing the robustness of productivity determinations from West-Coast Pacific ocean perch stock assessment data. N. Am. Fish. Manage. 22 301-310.

IWC (International Whaling Commission).

1992. Report of the Fourth Comprehensive Assessment Workshop on Management Procedures. Rep. Int. Whal. Comm. 42:305-321.

2001. Report of the standing working group on the development of an Aboriginal subsistence whaling management procedure. J. Cetacean Res. Manag. 4(suppl.):148-177.

Jakobsen, T.

1993. The behaviour of [F.sub.low] [F.sub.med] and [F.sub.high] in response variation in parameters used for their estimation. In Risk evaluation and biological reference points for fisheries management (S. J. Smith, J. J. Hunt, and D. Rivard, eds.), p. 119-125. Can. Spec. Publ. J. Fish. Aquat. Sci. 120.

Methot, R. D.

2000. Technical description of the stock synthesis assessment program, 56 p. NOAA Tech. Memo., NMFS-NWFSC-43.

Patterson, K. R., and G. P. Kirkwood.

1995. Comparative performance of ADAPT and Laurec-Shepherd methods for estimating fish population parameters and in stock management. ICES J. Mar. Sci. 52: 183-196.

Punt, A. E.,A. D. M. Smith, and G. Cui.

2001. Review of progress in the introduction of management strategy evaluation (MSE) approaches in Australia's South East Fishery. Mar. Freshw. Res. 52:719-726.

2002. Evaluation of management tools for Australia's South East Fishery. 2. How well do commonly-used stock assessment methods perform? Mar. Freshw. Res. 53:631-644.

Restrepo, V. R., and J. E. Powers.

1999. Precautionary control rules in US fisheries management: specification and performance. ICES. J. Mar. Sci. 56:846-852.

Rosenberg, A. A., and S. Brault.

1993. Choosing a management strategy for stock rebuilding when control is uncertain. In Risk evaluation and biological reference points for fisheries management (S. J. Smith J. J. Hunt, and D. Rivard, eds.), 243-249. Can. Spec. Publ. J. Fish. Aquat. Sci. 120.

Sampson, D. B., and Y. Yin.

1998. A Monte Carlo evaluation of the stock synthesis assessment program. In Fishery stock assessment models (F. Funk, T. J. Quinn II, J. Heifetz, J. N. Ianelli, J. E. Powers J. F. Schweigert, P. J. Sullivan, and C. I. Zhang, eds.), p. 315-338. Alaska Sea Grant, Fairbanks, AK.

Starr, P. J., P. A. Breen, R. H. Hilborn, and T. H. Kendrick.

1997. Evaluation of a management decision rule for a New Zealand rock lobster substock. Mar. Freshw. Res. 48: 1093-1101.

Andre E. Punt

School of Aquatic and Fishery Sciences University of Washington

1122 NE Boat Street Seattle, Washington 98195-5020

E-mail address: [e-mail]

Manuscript approved for publication 24 April 2003 by Scientific Editor.

Manuscript received 26 June 2003 at NMFS Scientific Publications Office.