A CONTRIBUTION TO A RESEARCH STRATEGY ON CHILDREN'S LIFE OUTCOMES.

Abstract

This paper is a contribution to the development of a strategy for cross-portfolio social science research on children's outcomes in New Zealand. Eight areas for research on the life chances of children are identified, and criteria for prioritising research projects are suggested. Capability for undertaking such research in New Zealand is also considered, and three major capability issues are explored.

INTRODUCTION

Improving the life chances of children is a primary focus of social policy in New Zealand. It is also one of the most challenging areas of public policy, because the factors influencing children's outcomes are multiple, complex and interacting. Identifying which factors are amenable to government influence, and how they can best be influenced, is extremely difficult.

Policy design and programme delivery should nevertheless proceed to the extent possible on the basis of well-informed judgements about what is most likely to achieve stated social objectives in a cost-effective way. The government invests large sums in income support for families, and on services intended to support families and improve outcomes for children. These are critically important investments in our national welfare. Relevant applied research in this area can be very valuable in informing the choice of public investments in children's life outcomes.

This paper is a contribution to the development of a strategy for cross-portfolio social science research on children's outcomes in New Zealand. The primary component of the paper is the specification of a small number of suggested topics as candidates for literature review. The topics are considered to be of particular importance for the design of social policy in New Zealand, and to reflect the key issues and questions being explored in research on the family and outcomes for children. Each priority area is presented with some illustrative research questions.

A secondary component of the paper is a discussion of issues relating to capability in New Zealand for undertaking research on children and families.

The paper is composed of four main sections.

In the first section, we explain why we believe that research should be focused on life outcomes for children, and locate our contribution within an overall framework for a research strategy. The second section contains a discussion of the role of research in designing effective social policies, and suggested criteria for ranking alternative research activities. The third section contains eight topics proposed as priorities for literature reviews, and the fourth section offers a discussion of issues relating to the capability for social science research in New Zealand.

FRAMING A RESEARCH STRATEGY ON CHILDREN'S OUTCOMES

Research on children's outcomes can be approached using the child as the unit of analysis, or the family as the unit of analysis. Using the family as the unit of analysis is likely to cause problems in specifying research questions, since it is likely to be impossible to develop an agreed definition or set of definitions of family.

Consider the goals of the social policy initiatives that this research is intended to inform. The main goal is improved outcomes for children, especially those who are disadvantaged. In our search for research themes, our attention was drawn to a number of reviews of policy-relevant international research. Many of these have titles containing phrases like "children's attainments", "the future of children", "child development", and "children's life chances". The emphasis in this body of research is on disadvantage.

Improved family functioning, or strengthened families, per se, is not the goal. Of course, what goes on inside families is of prime importance in influencing children's outcomes. The strategy of strengthening families is a means for working toward the end of better outcomes for children and young people.

There is a risk that a focus on improving outcomes for children may exclude efforts to improve outcomes for adult family members. We feel that this risk is not large. There is a close interaction between adult circumstances and outcomes for their children; inevitably action to improve children's outcomes will often involve improvement of the circumstances of parents or other caregivers. Moreover, studies based on the child as the unit of analysis offer a good way to discover the presence of problems faced by adult members of the household.

We suggest therefore that in its investment in cross-portfolio social science research government should place priority on research on life outcomes for children, with a focus on the following issues:

* What leads to poor life outcomes for children in areas such as health, education, employment, dependence on the state, and criminal offending?

* How influential are factors like income, welfare dependence, family formation and dissolution, other aspects of family functioning, and neighbourhood on these outcomes?

* How do the timing and duration of events and conditions affect outcomes?

* How can/does the state affect these factors?

A Suggested Framework for a Research Strategy

We consider it useful to place this exercise in the broader context of the development and implementation of a social science research strategy, and sketch out below seven stages in such a strategy. The process is not entirely linear; some stages would be done partly in tandem and there will be some iteration:

I. Selection and specification of the broad area of research

We believe the focus should be on children's life outcomes.

II. Selection and specification of priority topics or research questions within the broad area

This has been done in the section on "Priority Areas", below. These topics were developed by considering strategic social policy priorities, by incorporating input from policy departments on their knowledge requirements, and by some preliminary examination of the literature.

III. Reviews of past and current work for each of the priority topics

These reviews should include, but not be confined to, literature reviews. A "stocktake" of New Zealand activity in the area is also needed, including descriptions of existing New Zealand data sets and current data-gathering efforts, research projects underway, and so on.

IV. Development of a menu of potential research projects

The heart of each research project should be a well-specified research question. Needs for data and skills, and opportunities for "piggybacking" on other studies or surveys should be identified. A project could be the development of a data set useful for answering a range of questions.

V. Assessment of capability within New Zealand for undertaking different types of research projects

We discuss capability issues in the section on "Capability", below.

VI. Mapping the capability to the menu of potential projects, in order to select a feasible subset, subject to funding constraints 

VII. Ranking the subset of feasible projects

We propose criteria for ranking in the second part of the next section.

PRIORITISING SOCIAL POLICY RESEARCH

Examination of strategic social policy priorities identified by agencies reveals the major concern to be the existence and possible growth of a small proportion of families suffering chronic multiple disadvantage. These families are seen to be consuming a highly disproportionate share of resources in the welfare, health, education and justice sectors. Many of these families are reliant on the state for income support, and there is concern about the impact of this on the life chances of their children. Maori and Pacific ethnic groups are highly over-represented in this group.

The Role of Social Policy Research

How can social policy research contribute to the design of social policies that will improve the outcomes for these children and their families?

Social policy is unavoidably based on assumptions and beliefs about the nature of society and social processes, and the causes of social outcomes. A fundamental role for a social science research strategy is to test and refine the assumptions on which social policies are based. At some risk of over-generalisation, the following are examples of key assumptions that appear to underlie the design of recent policies in the area of children's outcomes, and which could be tested with empirical research.

* There is a significant incidence of inter-generational transmission of chronic multiple disadvantage.

* Government interventions can break cycles of disadvantage and dependence.

* Tightly targeting assistance on those families most in need is generally more effective and efficient than more universal approaches.

* Better delivery of services to Maori is the key to better outcomes for Maori.

* Families, rather than individuals or communities, should be the focus for government interventions.

* Reliance on state-provided income is not desirable, and can harm as well as help the recipients.

Another "high-level" role for research is the illumination of tensions between different priorities in some situations, in order to judge their relative importance. One example is a focus on strengthening the family in the small minority of cases where the family is the main risk to better outcomes for the children. Another example is a focus on the importance of adults being in paid work where this could lead to inadequate time available for the needs and development of young children.

Criteria for Selecting and Prioritising Research Projects

Criteria for selecting and prioritising research efforts are needed both for research topic areas and for specific research projects. Of the following seven criteria, the first two, "significance for national welfare" and "broad policy relevance", apply to the prioritising of the research topic areas. The subsequent five criteria apply to the selection of research projects.

* Likely significance for national welfare

This requires judgements about the magnitude of social costs that might in principle be avoidable, and the likelihood that feasible interventions can reduce these costs.

* Policy relevance in a broad sense -- the degree to which the research tests the assumptions and improves the analytic basis of key public policies and interventions

Research that detects and explores causal relationships is vital. Policy-relevant research will generate results in areas where government is capable of influencing outcomes, and where there is wide support for government playing an active role in intervening to alter outcomes.

* Management of risk

Given the complexity of causal relationships in this area, a mixed strategy, involving the generation and use of multiple data sources and methodologies, is desirable. A mixed strategy is also likely to involve efforts with different time profiles of benefits -- potentially high-return but longer-term approaches should be adopted along with faster payback approaches. A further risk management strategy is to anticipate pressures for the rapid expansion of new, apparently promising but not yet evaluated programmes or policy approaches. Anticipating such pressures and responding by initiating rigorous research and evaluation can reduce the risk of large policy errors.

* Policy relevance in a narrow sense -- the degree to which the proposed project will generate results that can be used to affect children's outcomes of greatest concern

What kinds of policy tools might be available to make use of the results of the research? Are they politically feasible? Which positive and negative outcomes are we most concerned about?

* Tractability -- the likelihood that the research project is feasible and that the methodological approach used to analyse the data will generate useful results

* Capability -- the required skills, data and institutional support are available

* Cost-effectiveness -- the ratio of expected costs to expected increase in knowledge and understanding

Some projects may generate benefits that will contribute to the value of, or make possible, other projects; such external benefits and option value should be kept in mind when assessing research "bang for the buck".

PRIORITY AREAS

We have tentatively identified eight areas in which greater knowledge and understanding of influences on the life chances of children could improve the design of cross-portfolio social policy in New Zealand. Each priority area is presented with some illustrative research questions.

Throughout the process of selecting the priority areas we kept in mind the strategic priorities evident in contemporary social policy. We have also taken into account our two criteria for selecting priority research areas -- namely, significance for national welfare, and policy relevance in a broad sense.

Our first step was to scan the academic social policy literature on children's outcomes and family dynamics. We then examined the statements made about knowledge needs by departments during the Royal Society study (Royal Society 1998). Concurrently, we conducted a number of interviews with social policy advisers and researchers. Sample research questions in each area were generated through some further review of the literature and interaction with Mary Corcoran (Professor of Social Work at the University of Michigan).

If work is to be done on any of these areas, we envisage three components in the next stage:

* Extraction of "stylised facts"(2) from international and New Zealand literature;

* Identification of the current burning research questions and the methodologies being used to address them; and

* A survey of research and data-gathering currently underway or planned in New Zealand.

Three types of research are needed for social policy(3):

Type 1 Basic descriptive research on the whole population;

Type 2 The uncovering of causal relationships; and

Type 3 Evaluation of policies and interventions.

Our organisation of priority research areas is based broadly on these three research types. It follows that the order in which these research areas are presented does not imply anything about their relative importance.

The first two research areas that we have identified -- "Entrenched Disadvantage" and "Social Indicators" -- call for Type 1 research, that is, descriptive research needed for the identification and scoping of current and emerging problems.

The next four research areas -- "How Much Does Income Explain?" "Mothers in Paid Work" "Neighbourhood Effects" and "Functioning of Families" -- call for Type 2 research, that is, the study of causal relationships. Many factors are statistically associated with poor outcomes for children. However, changing one of these factors will not necessarily change outcomes; a correlation does not imply a simple causal relationship. Agreement about the strength and form of causal relationships evolves slowly as evidence accumulates from many studies. Each of these priority areas is focused on a set of factors that may "explain" children's outcomes and are particularly relevant to New Zealand's social policy design.

The last two research areas -- "Impacts of Policy on Family Relationships" and "What Programmes Work and Why?" -- call for Type 3 research, that is, these research areas are concerned with the effects of government interventions.

Area 1: Entrenched Disadvantage?

A basic assumption underlying the current emphasis on targeting is that a subset of the population with a concentration of bad outcomes is developing and becoming entrenched. In New Zealand, the focus is on the children and young people who are seen as being at high risk of poor life outcomes. High-risk families are defined as families who are experiencing multiple and persistent disadvantages that compromise family functioning, and which increase the chances that children and young people will have poor long-term outcomes. It is important, for ensuring good policy design, that the characteristics of the most disadvantaged subset of the population are well understood.

* Is there a subset of the population in New Zealand experiencing multiple and persistent disadvantage? What are the defining characteristics of this group? How much is this group confined to families with children? Is level of income, duration of low income, or the presence of multiple problems the most important defining characteristic?

* What are the recent historical trends in the size and composition of this subset?

* How does the focus in New Zealand on high-risk families compare with attempts in other countries to define a subset of the population (sometimes known as an "underclass") as a primary focus for social policy? What lessons emerge from the international literature on the defining characteristics and key attributes of such population subsets?

* What ethnographic work exists in New Zealand that throws light on the characteristics and functioning of high-risk families and the situation of

children in them?

* What information exists on entry and exit rates, and distribution of residence times? Is there evidence of inter-generational transmission of multiple persistent disadvantage?

* To what extent is this group geographically concentrated in certain regions and/or neighbourhoods/districts? What are the key characteristics of any geographic concentrations in terms of indicators such as unemployment, crime, poverty and ethnicity?

* What studies exist of the aggregate monetary costs for society of family dysfunction and poor life skills for children?

Area 2: Social Indicators

Careful selection of indicators is important for monitoring trends at a high level, in order to describe broad social outcomes, identify emerging and potential social problems, and understand their magnitude and direction of change. They are likely to have little value as measures of the effectiveness of individual programmes; this is the focus of research area 8. (Because of their high level, such indicators are the outcome of the complex interplay of a variety of social processes.) In New Zealand, a set of indicators of outcomes and risk behaviours for children and young people has been adopted (Interdepartmental Report 1997, 1998, 1999). Indicators of outcomes have also been developed for assessing disparities between Maori and non-Maori (Te Puni Kokiri 1998, 2000). The selection of a good set of outcome indicators is a very difficult task, and a review of New Zealand and international experience in this area should be a priority.

* What high-level social indicators are proposed and discussed in the international literature?

* What key indicators have been used in other countries to measure outcomes for children at a high level? What indicators have been used for material hardship, housing conditions and inadequate nutrition? How have these indicators been used to inform policy design? What value have they added?

* What is the best set of indicators for capturing disparity in social and economic well-being between Maori and non-Maori?

Area 3: How Much Does Income Explain?

A significant proportion of total government expenditure goes into transfer payments to families with children. In effect, the major strategy for strengthening families is the provision of a basic level of income. There are important questions to answer about the explanatory power of family income on children's outcomes.

* How is family income correlated with children's educational participation and achievement, health, developmental outcomes and rate of criminal offending?

* How does children's income correlate with their parents' income, and with that of their siblings?

* At what ages are children most affected by the level of family income? How do sudden drops in income affect children?

* What is the evidence about the effects of persistent versus short-term low income?

* How much variation in outcomes is "explained" by family functioning, controlling for other variables like income and socio-economic status?

* Is there evidence that earned income has a more beneficial effect on children's outcomes than welfare income?

* How is children's cognitive development affected by poor nutrition?

Area 4: Mothers in Paid Work(4)

A major social change in New Zealand is the dramatic increase in the proportion of mothers who are in the paid workforce. The issues associated with the tension between the demands of paid work and the care and development needs of children are generally greatest for women who are sole parents.

* What is the effect, on children's educational and social outcomes, of their mothers being in paid work?

* How do the effects of mothers' employment on children's outcomes differ between one-parent and two-parent households, and how do these effects vary by the age of the children, and the nature of childcare arrangements (both pre-school and out-of-school care)? How do job conditions (such as pay levels, number and regularity of hours worked and the nature of the job) affect child outcomes?

* Does part-time employment by mothers tend to lead to full-time employment when children become independent? Do wages increase over time for women who work part-time or part-year? Do wages grow with experience for women with low levels of schooling and skills? Do wages grow for women with histories of receipt of social welfare benefits?

* What barriers to work do low-income mothers face, in addition to the usually identified ones of childcare, little schooling and lack of work experience?

Area 5: Neighbourhood Effects

An active area of social research internationally is the influence of neighbourhoods on families and children. If significant, this would suggest including neighbourhood effects as a criterion for evaluating social policies.

* Does living in a particular neighbourhood affect children's outcomes, after controlling for socio-economic status?

* How much of the variation in children's outcomes is driven by neighbourhoods? Are neighbourhood effects non-linear? For example, are neighbourhood effects not significant unless the neighbourhoods are very disadvantaged?

* What proportion of children at greatest risk of poor outcomes live in disadvantaged neighbourhoods?

* What specific characteristics of neighbourhoods drive outcome effects? What are the individual and family relationships with external institutions that have significant impacts on children's outcomes? Under what conditions are these impacts likely to be positive/negative?

Area 6: Functioning of Families

Interactions within families are of prime importance in determining outcomes for children. Much of our understanding of family interactions is based on the stable nuclear family structure that has become increasingly uncommon. If families are to be strengthened, we need to know what is going on inside them.

* What parental/caregiver behaviours are the critical ones for good child outcomes? With respect to parents, how important is quality of interactions versus quantity of time spent with children?

* Are there particular times or transitions when the quality of family interactions is especially crucial? At what ages are children most vulnerable to stresses such as family break-up or conditions such as prolonged period on benefit?

* How and why do marital transitions affect children?

* What does the literature suggest about whether the father's presence matters, and why? Does this vary with the gender of the child? Do the frequency and/or nature of interactions between non-custodial fathers and children affect child outcomes? Does receipt of money for child support have more positive effects on child outcomes than other sources of money?

* How, and how much, does domestic violence between adults affect child outcomes? How significant are the effects of child abuse on child outcomes? Does this vary with characteristics such as age and gender of child, and nature of abuse?

* Does the immersion in and knowledge of Maori language and culture affect the outcomes of Maori children?

* Who are the other household members in multiple/extended family households in New Zealand and what roles do they play? How does their presence impact on outcomes for children?

Area 7: Impacts of Policy on Family Relationships

The demography and functioning of families is greatly influenced by policies and legislation governing legal, financial and custodial relationships. Legislation regulating family relationships is based on a complex mixture of factors, including legal rights and expected effects on adults and children. However, the laws governing relationships between parents can have unintended impacts on children. Therefore, it is important to develop a better understanding of the effects of different approaches to regulating family relationships.

* What are the key ways in which government policies and legislation influence private decisions over family formation, dissolution and reformation? What are the impacts on family living arrangements and family types?

* What are the effects on family formation and dissolution of different legal rules applying to marriage and divorce? What effects do different legal rules concerning the division of matrimonial property have on the economic position of formerly married adults and their children? What effects do different legal rules applying to "relationships in the nature of marriage" have on decisions to enter and dissolve such relationships, and on the economic position of former partners and their children?

Area 8: What Programmes Work, and Why?

The clearest message that we received from policy advisers was that they wanted to know what works. Social programmes are certainly being evaluated. However, a major concern is that evaluation is usually focused on process and outputs, and is short term. There is little evaluation of outcomes. Another concern is that knowledge of successful and unsuccessful programmes both in New Zealand and overseas is held by individuals who take their knowledge with them when they leave the job.

* What are the key design features associated with the most successful (effective and cost-effective) programmes? What are the key design features associated with the unsuccessful programmes? What instruments are used for measuring success -- both short run and long run? Where such programmes include a bundle of services, can the effects of different programme elements be separated out?

* Is there evidence that some types of interventions are harmful?

* What are the key elements in the broader environment in which highly successful or unsuccessful programmes are nested?

* Is provision of services by providers of the same ethnicity systematically associated with better outcomes for the families and children concerned?

* Is intensive intervention at certain stages of a child's life effective?

* What is the optimal balance between highly targeted and less targeted interventions aimed at improving outcomes for children? What screening instruments should be used for targeting?

* What are the most effective models for delivery of cross-portfolio services to high-risk families? One class of delivery models is partnerships between different levels of government and the voluntary sector. Another class is case management models with the case manager acting as a broker for all services to a single family, perhaps as a budget-holding case manager. Under what conditions are services best delivered to children outside the home instead of being delivered to children and parents in the home?

CAPABILITY

In our suggested framework for developing a research strategy, we noted that an assessment of capability was required for reviews of past and current work for the priority research topics (stage III), as well as for the actual undertaking of specific research projects (stage V). We begin by assessing capability for stage III, and follow it with some preliminary work on capability for stage V. The latter consists of an overview and discussions of three particular capability issues.

Capability for Undertaking Reviews of Work in Priority Areas

For stage III of developing a research strategy we see the need for three forms of expertise:

* Familiarity with international literature

This will include knowledge of key stylised facts, good overviews, key figures in the field, current research emphases, assessment of feasibility of research and "bang for the buck". This expertise is most likely to be found in one or two key international researchers in the field, whose own research has a focus on policy.

* Ability to understand the literature, and summarise the results

It will be cost-effective in the short term and build up local expertise in the long term, if this is done by people based in New Zealand. In the next section, we expand on what we mean by "understand the literature".

* "Local" New Zealand knowledge of existing policy, research and data

Review of existing data sets is especially important since the collection of new data is expensive. If an empirical research question can be answered by using existing data, then this should be done.

Employing top international social policy researchers to identify seminal papers, the best overviews and the current "hot" research questions, and to interact with New Zealand-based researchers, is highly cost-effective. At the same time, policy issues, the assumptions on which policies are based, the institutions and legislation are all to some extent country-specific. Besides providing a New Zealand perspective on an international literature review, New Zealand-based reviewers of priority areas can review New Zealand literature, consult local experts, and locate and describe existing New Zealand data sets and current data-gathering efforts. Some capability will be developed in New Zealand in the process.

Another reason for developing New Zealand capability in literature review is that such review should be ongoing. Both individual and institutional capability is required for a watching brief to be kept on relevant international literature.

The result of the review stage of the process may in some instances reveal that a question has been well studied, and there is no reason to think that the results would not transfer reasonably well to New Zealand. In such a case, the problem is not a lack of relevant research, but may be a lack of dissemination of research. The dissemination of the results of research may be seen as peripheral to a research strategy, but it is a large part of closing the knowledge gaps of policy makers. The expertise required for dissemination of research involves the ability to both understand the literature and communicate the key results clearly. Both forms of expertise must be found in the same person. Researchers may not be the best people to disseminate their own research(5).

Assessment of Capability in New Zealand for Undertaking Research Projects

Capability for undertaking research projects in New Zealand (stage V of developing the research strategy) has three dimensions -- skilled researchers, data, and ongoing institutional support for both. To these three we could add funding, but suggest that this is best incorporated as a budget constraint on the selection of projects in stage VI.

Research capability requirements depend upon the type of research proposed. Recall that our three groups of priority areas correspond approximately to the three types of research described by Mayer -- descriptive research on the entire population, research that tries to determine causal relationships, and evaluation research. These three are all needed for choosing government interventions that will improve the outcomes of children and young people. An "ideal" social policy research project would link all three along the following lines:

* Identify problems that are large and increasing, using basic descriptive research on the entire population;

* Define outcomes of interest, and establish proxies for their measurement;

* Form hypotheses about influences on these outcomes by drawing on any relevant quantitative research and qualitative research like case studies and surveys;

* Be alert for policy changes that might provide the opportunity for "natural experiments";

* Look for statistical associations between potentially explanatory variables and the outcome proxies;

* Postulate and test causal models that reflect plausible "stories";

* Use judgement about the results of these tests along with the weight of evidence from similar studies to infer stylised facts about true causal relationships;

* Select causal factors whose effect on outcomes is both statistically significant, and significant in a policy or practical sense;

* Establish a menu of interventions with the potential to change these causal factors; Choose promising interventions -- those with characteristics associated with success; and

* Experiment with these interventions, using randomised controlled trials. If randomisation is not possible, use some other design technique to control for confounding factors.

It would be possible, albeit tedious, to write a list of the capability requirements that would cover this whole span of social science research. The list of research skills would look like the course outlines for doctoral-level courses in public policy, sociology and economics at top universities. The data sets envisaged would be prohibitively expensive pipe dreams. Institutional support would be generous and stable. We turn from the ideal to the realistic and focus on three capability issues that seem particularly important.

Capability Issue: Longitudinal Studies of Children

Descriptive research in New Zealand on the whole population has been based on households or individuals. Since households (and families) change over time, panel data can only be collected on individuals. A deep understanding of the influences on children's outcomes can be gained by following individual children over time.

While longitudinal studies of children are extremely valuable, it is important to understand that the collection of such data is expensive, and that valuable insights can be gained from other forms of data. For example, hypothesis-driven analyses of cross-sectional data can be used to check some expected correlations and thus strengthen or weaken the evidence for particular causal relationships.

There have been two longitudinal studies of children in New Zealand -- the Fergusson study in Christchurch and the Silva study in Dunedin. These studies have been immensely valuable in identifying relevant variables, in constructing composite measures such as Fergusson's family difficulties score, and in establishing statistical associations between potential causal factors and children's outcomes. However, dissatisfaction with the inability to generalise from two South Island cities to the rest of the country is widespread. Serious consideration should be given to the initiation of another longitudinal study of children, in which random samples of children in different geographic regions are followed from birth to adulthood.

Statistics New Zealand has conducted a feasibility study for the collection of longitudinal data on approximately 20,000 individuals in 10,000 households. The focus is on the household and (nuclear) family income of individuals, but data would also be collected on employment, benefit receipt, and a number of standard-of-living variables. Individuals would be interviewed annually. For some variables, the plan is to elicit information on monthly changes. This study should yield a valuable longitudinal data set and enable investigation of questions about family dynamics and the family circumstances of children. For example -- what are the patterns of family formation and dissolution, and how do these patterns vary between different income, ethnic and age groups? Or, what proportion of families spend long periods of time in low-income states and for what proportion is it a transition period? However, this study is not a longitudinal study of children from birth to adulthood. The children in the study will be of different ages when the study begins and will be followed for only eight years -- the lifetime of the study. Given the high cost of collecting longitudinal data, there is a window of opportunity here that should be explored.

If a new longitudinal study of children is to be initiated in New Zealand, an overview of the design, sample size and cost-cutting techniques of major longitudinal datasets in other countries should be performed first. There will be valuable lessons to be learned from surveys such as the US National Longitudinal Survey of Youth, as well as from New Zealand's two existing longitudinal surveys of children.

Capability Issue: Skills for Studying Causal Relationships

Studying causal relationships relating to children's life outcomes requires quantitative research skills on two levels. The first level is that of the highly trained econometrician who is able to analyse panel data sets, and deal with problems like attrition. The second level is sufficient training in quantitative analysis for good understanding of the literature. In order to be able to understand research on causality, it is necessary to have some experience in analysing data, and have a good understanding of such basic concepts as correlation not implying causality, left-out variable bias, the difference between statistical significance and policy significance, and why the testing of a poor causal model might give a high correlation.

We do not wish to imply that there is no place for qualitative research. Certainly, econometric modelling has become the method preferred by many academics for studying causal relationships. Nevertheless, research that is useful for the development of social policy often differs from research that is interesting to "pure" social scientists. Academics are generally attracted to research questions that are tractable and allow for rigorous application of methodology. For policy development, the relevance of the research question is paramount, and it may often be possible to do only qualitative research.

Basic social science research in New Zealand will be limited by the lack of funds for large panel data sets (and the lack of a large population on which to collect such data). We see research on children's outcomes and family dynamics in New Zealand, that goes beyond description to determining causality, developing as follows:

* Develop hypotheses about causal relationships in New Zealand from qualitative and quantitative research in New Zealand and stylised facts from overseas literature;

* Check how well the stylised facts from overseas literature will "travel" to New Zealand by comparing trends, distributions of key variables and local knowledge. For instance, stylised facts about African Americans may not translate well to stylised facts about Maori -- the literature on Native Americans may be more relevant. We might have more confidence in stylised facts that hold across different ethnic groups. Although the United States is particularly rich in social policy research, studies from countries with policy cultures and institutions more similar to New Zealand -- like Australia, Canada and the United Kingdom -- should be sought; and

* Use cross-sectional data sets to see if variables are correlated in a way that is consistent with the hypotheses.

Capability Issue: Evaluation of Government Interventions

The evaluation of the outcomes of social programmes typically involves answering two questions:

* How sure are we that the intervention was effective?

* How effective was it?

To the statistical skills needed for answering these questions, economic skills should be added in order to answer a third question:

* How cost-effective was it?

In our discussions with various consumers of social policy, we found concerns about evaluation research as currently practised.

First, resistance to randomised, controlled trials of social interventions is widespread. More people need to understand why control of confounding variables is essential to assessing the effects of a programme. Randomisation is the easiest way to achieve this (although not the only way). Exploring evaluation methods that control for confounding variables, and which are politically acceptable in New Zealand, appears to be a strategically important element of developing capability for effective social science research in New Zealand.

Second, we were advised that evaluation studies conducted by government departments generally focus on operational aspects and acceptability of the interventions, rather than assessing changes in the outcomes of interest. This may reflect a number of factors, including cost, initial programme design, shortage of necessary skills, the cross-portfolio nature of many outcomes of interest, and a tendency for Ministers to focus the efforts of their departments on the urgent rather than the longer term. Responses to these problems include involving evaluation experts in the design of programmes or pilots, and having some outcome evaluation conducted by institutions other than the policy agencies.

CONCLUDING REMARKS

This report contributes to a strategy for developing a cross-portfolio social policy research agenda on children's outcomes/family dynamics in New Zealand. In the section on "Framing a Research Strategy on Children's Outcomes" we laid out a framework for this strategy. We return to this framework to clarify our contribution:

Stage I Selection and specification of the broad areas of research

In the section on "Framing a Research Strategy on Children's Outcomes" we explain our selection of the descriptor "Children's Outcomes".

Stage II Selection and specification of priority areas of research

We selected eight priority areas and specified each by a brief description followed by some illustrative (though not comprehensive) research questions. The questions are not tightly specified -- that is the task of the proposer of a research project. Note also that we have not ranked the eight priority areas.

Stage III Reviews of past and current work for each priority area

This is the next task in developing the strategy. Of course, our proposed priority areas need further consideration and ranking before any reviews are commissioned. We have assessed the capability required for this review stage.

Stage IV Development of a menu of potential research projects

Some of our illustrative questions on the eight topic areas could eventually evolve into potential research projects.

Stage V Assessment of capability in New Zealand for undertaking different types of research projects

We have begun this task in the section on "Capability".

Stage VI Mapping capability to the menu of potential projects to select a feasible subset

This task cannot be done yet.

Stage VII Ranking the subset of feasible projects

We have proposed criteria for ranking in the section "Prioritising Social Policy Research".

The task of improving life outcomes for children, to the extent that it is amenable to government action, cannot be achieved by one single government department. Children at risk and their families require support of many kinds, and interdepartmental cooperation is essential for such support to be delivered effectively. Cross-portfolio social policy research is required for providing the knowledge base for designing and delivering effective support.

Murray Petrie(1)
Principal
The Economics and Strategy Group

Janice Wright
Analyst and Consultant

(1) Acknowledgements

This paper is based on a report commissioned by the Ministry of Education in 1999 to help develop a strategy for cross-portfolio social science research on family dynamics, following the selection of this area by the Officials Working Group on Applied Social Science (OWGASS) as the top priority for social policy research (Petrie and Wright 1999). During the course of the project, expert advice was provided by Mary Corcoran, Professor of Social Work at the University of Michigan. The draft report was reviewed separately by Professor Corcoran and by Professor Susan Mayer of the Harris Graduate School of Public Policy Studies at the University of Chicago. Helpful comments were also provided by David Wood and Susanna Roddick. The list of references contains many sources that, while not cited in the text, broadly informed the paper. They are included as useful references for any reader who may wish to read further in the area.

Correspondence

Murray Petrie: [e-mail]

Janice Wright: [e-mail]

(2) Stylised facts are major messages emerging from the literature. For example, three stylised facts on the relationship between childhood poverty and children's life chances in the United States taken from Duncan et al. (1998) are:

* Family income has much stronger associations with achievement and ability-related outcomes for children than with measures of health and behaviour;

* Early childhood appears to be the stage in which family economic conditions matter most; and

* The impact of family income on completed schooling is larger for children in low-income families than for those in high-income families.

(3) This is Susan Mayer's three-way categorisation of social policy research as outlined in her review of the Royal Society report, "Family dynamics: a research strategy prepared for the Ministry of Research, Science and Technology, 11 December 1998".

(4) The term "Mothers in Paid Work" is shorthand for the situation where the primary carer is in the paid workforce; of course this is not always, although it is usually, the mother.

(5) "Researchers often over- or under-state their results (depending on their own proclivities) and are often too steeped in the language of their discipline to communicate well to a wider audience." (Mayer 1998:7)

REFERENCES

Boggess, Scott and Mary Corcoran, with Stephen P. Jenkins (1999) Cycles of Disadvantage? Institute of Policy Studies, Victoria University of Wellington.

Davey, Judith (1998) Tracking Social Change in New Zealand: From Birth to Death iv, Institute of Policy Studies, Victoria University of Wellington.

Duncan, G.J., J. Brooks-Gunn, W.J. Yeung and J.R. Smith (1998) "How much does childhood poverty affect the life chances of children?" American Sociological Review, 63:406-423.

Haveman, Robert and Barbara Wolfe (1995) "The determinants of children's attainments: a review of methods and findings" Journal of Economic Literature, 33:1829-1878.

Interdepartmental Report (1997) "Strengthening Families strategy: report on outcome measures and targets" Wellington.

Interdepartmental Report (1998) "Strengthening Families strategy: report on outcome measures and targets" Wellington.

Interdepartmental Report (1999) "Strengthening Families strategy: report on outcome measures and targets" Wellington.

McLoyd, V. (1998) "Socioeconomic disadvantage and child development" American Psychologist, 53(2):185-204.

Ministerial Review of Applied Social Science Capacity (1995) "Drawing on the evidence: social science research and government policy" report to the Government through the Minister of Research, Science and Technology, Wellington.

Mayer, Susan (1997) What Money Can't Buy: Family Income and Children's Life Chances, Harvard University Press, Cambridge, Massachusetts.

Mayer, Susan (1998) "Review of Royal Society Report" Ministry of Education, unpublished.

Moffitt, Robert (1996) "The effect of welfare on marriage and fertility: what do we know and what do we need to know?" Department of Economics and Department of Population Dynamics, Johns Hopkins University, Baltimore.

National Science Foundation (1999) "Investing in human resources: a strategic plan for the Human Capital Initiative" http://www.nsf.gov/sbe/sber/sociol/ works1.htm.

Newman, Katherine (1999) No Shame in My Game: The Working Poor in the Inner City, Vintage Books, New York.

Petrie, M., and J. Wright (1999) "A contribution to a research strategy on children's life outcomes", report commissioned by the Ministry of Education, Wellington.

Royal Society (1998) "Family dynamics: a research strategy prepared for the Ministry of Research, Science, and Technology, 11 December 1998" The Royal Society of New Zealand, Wellington

Shirley, Ian, P. Koopman-Boyden, I. Pool and S. St.John (1997) "Family change and family policies in New Zealand" in S. Kamerman and A. Kahn (eds.) Family Change and Family Policies in Great Britain, Canada, New Zealand and the United States, Clarendon Press, Oxford.

Te Puni Kokiri (1998) "Progress towards closing social and economic gaps between Maori and non-Maori", report to the Minister of Maori Affairs, Wellington.

Te Puni Kokiri (2000) "Progress towards closing social and economic gaps between Maori and non-Maori", report to the Minister of Maori Affairs, Wellington.