Identification problems in the social sciences and everyday life. (Association Lecture).

1. Introduction

The Reflection Problem

Here is an identification problem from everyday life: Suppose that you observe the almost simultaneous movements of a person and of his image in a mirror. Does the mirror image cause the person's movements, does the image reflect the person's movements, or do the person and image move together in response to a common external stimulus? Empirical observations alone cannot answer this question. Even if you were able to observe innumerable instances in which persons and their mirror images move together, you would not be able to logically deduce the process at work. To reach a conclusion requires that you understand something of optics and of human behavior.

A like inferential problem, which I have called the reflection problem (Manski 1993a), arises if you try to interpret the common observation that individuals belonging to the same group tend to behave similarly. Two hypotheses often advanced to explain this phenomenon are endogenous effects, wherein the propensity of an individual to behave in some way varies with the prevalence of that behavior in the group; and correlated effects, wherein individuals in the same group tend to behave similarly because they face similar environments and have similar individual characteristics.

Similar behavior within groups could stem from endogenous effects (e.g., group members could experience pressure to conform to group norms) or group similarities might reflect correlated effects (e.g., persons with similar characteristics might choose to associate with one another). Empirical observations of the behavior of individuals in groups, even innumerable such observations, cannot per se distinguish between these hypotheses. To draw conclusions requires that empirical evidence be combined with sufficiently strong maintained assumptions about the nature of individual behavior and social interactions.

Why might you care whether observed patterns of behavior are generated by endogenous effects, by correlated effects, or in some other way? A good practical reason is that different processes have differing implications for public policy. For example, understanding how students interact in classrooms is critical to the evaluation of many aspects of educational policy, from ability tracking to class size standards to racial integration programs.

Suppose that, unable to interpret observed patterns of behavior, you seek the expert advice of two social scientists. One, perhaps a sociologist, asserts that pressure to conform to group norms makes the individuals in a group tend to behave similarly. The other, perhaps an economist, asserts that persons with similar characteristics choose to associate with one another. Both assertions are consistent with the empirical evidence. The data alone cannot reveal whether one assertion or the other is correct. Perhaps both are. This is an identification problem.

Identification and Statistical Inference 

Identification problems are problems of deductive logic. The conclusions that a researcher can logically draw are determined by the assumptions and data that are brought to bear. The available data about human behavior are typically limited, and the range of plausible assumptions is wide. So researchers who analyze the same data under different maintained assumptions may, and often do, reach different logically valid conclusions.

Empirical researchers often ask econometricians for assistance in "solving" identification problems. This is asking too much. What econometricians can usefully do is to clarify what conclusions can and cannot logically be drawn given empirically relevant combinations of assumptions and data.

For more than a century, methodological research in the social sciences has made productive use of probability and statistics. One supposes that the empirical problem is to infer some feature of a population described by a probability distribution and that the available data are observations extracted from the population by some sampling process. One combines the data with assumptions about the population and the sampling process to draw statistical conclusions about the population feature of interest.

Working within this familiar framework, econometricians have found it useful to separate inference into statistical and identification components. Studies of identification determine the conclusions that could be drawn if a researcher were able to observe a data sample of unlimited size. Statistical inference seeks to characterize how sampling variability affects the conclusions that can be drawn from samples of limited size.

Identification and statistical inference are sufficiently distinct for it to be fruitful to study them separately. The usefulness of separating the identification and statistical components of inference has long been recognized. Koopmans (1949, p. 132) put it this way in the article that introduced the term identification into the literature:

In our discussion we have used the phrase "a parameter that can be determined from a sufficient number of observations." We shall now define this concept more sharply, and give it the name identifiability of a parameter. Instead of reasoning, as before, from "a sufficiently large number of observations" we shall base our discussion on a hypothetical knowledge of the probability distribution of the observations, as defined more fully below. It is clear that exact knowledge of this probability distribution cannot be derived from any finite number of observations. Such knowledge is the limit approachable but not attainable by extended observation. By hypothesizing nevertheless the full availability of such knowledge, we obtain a clear separation between problems of statistical inference arising from the variability of finite samples, and problems of identification in which we explore the limits to which inference even from an infinite number of observations is suspect.

My Research Program

I have been concerned with identification problems throughout my career. My early research concerned the problem of inference on people's preferences from observations of the choices that they make. Economists are fond of saying that choice behavior "reveals preferences." In fact, observation of the action that a person chooses only reveals that this action is weakly preferred to all other feasible actions. It does not reveal how the person ranks nonchosen actions relative to one another. Discrete choice analysis, as it is practiced in econometrics, combines data on choices with assumptions about the decision rules that individuals use when making the choices that researchers observe. The concern of my early research was to determine what can be learned about preferences given data on choices and relatively weak assumptions about the decision rules that people use. This is an identification problem.

Over time, I have come to think that, although statistical problems contribute to the difficulty of empirical research, identification is the more fundamental problem of the social sciences. In what follows, I first describe the broad themes of a research program that I began in the late 1980s and continue today. I next show how these themes have played out in my analysis of the selection problem, a fundamental and pervasive identification problem. I then examine how the selection problem manifests itself in the econometric analysis of market demand.

2. Broad Themes

My 1995 book, Identification Problems in the Social Sciences (Manski 1995), puts forward four broad, related themes. They are as follows.

Begin with the Data Alone

The prevalent approach to empirical research in the social sciences begins by maintaining assumptions that are strong enough to identify quantities of interest and to yield statistically precise point estimates of these quantities. Concerns about the credibility of assumptions are commonly addressed through the performance of specification tests and/or sensitivity analysis. Concerns about credibility may also be addressed by exploring how estimates change and statistical precision falls as functional form and distributional assumptions are weakened.

A complementary approach to empirical inference begins by asking what can be learned from the data, given only the knowledge of the sampling process and no other prior information. Having determined this, one may then ask what more can be learned given successively stronger forms of prior information. This approach yields a series of successively tighter bounds on quantities of interest. The bound is widest when no assumptions are maintained, and it narrows as stronger assumptions are imposed. Sufficiently strong assumptions narrow the bound to a point.

This approach to empirical research has particular value when social scientists invoking different strong assumptions find themselves in disagreement about the interpretation of empirical evidence. Establishing the conclusions that hold up under weak assumptions can build a domain of consensus and confine disagreements to those questions whose resolutions really do require controversial maintained assumptions.

Points and Bounds

I have just made reference to bounds on quantities of interest. Social scientists commonly think of identification as a yes-or-no question: a parameter is either identified or not identified. Yet identification, generally, is not a binary state. A researcher who does not have rich enough prior information and data to infer the exact value of a parameter may nevertheless be able to partially identify the parameter--that is, to bound it.

The fixation of social scientists on point identification has inhibited the appreciation of the potential usefulness of bounds. I use the term "fixation" because I cannot readily understand the scientific basis for the longstanding notion that a parameter is either identified or not. Bounds on parameters have been reported from time to time in the methodological literature. Nevertheless, in empirical research and in the teaching of econometrics, identification has generally been thought of as point identification.

Coping with Ambiguity

The scientific community tends to reward researchers who produce strong findings, and the public tends to reward those who make unequivocal policy recommendations. These incentives tempt researchers to maintain assumptions that are far stronger than they can persuasively defend, to draw strong conclusions. We need to develop a greater tolerance for ambiguity. We must face up to the fact that we cannot answer all of the questions that we ask.

My research on identification has yielded a number of formal negative results. I have reported simple "impossibility theorems" showing that broad classes of hypotheses are not empirically testable unless sufficiently strong assumptions are maintained. I have found that researchers are often reluctant to acknowledge that, given the available data, they can logically draw some conclusion of interest only if they maintain strong assumptions that may have limited credibility. Be that as it may, I feel strongly that both positive and negative findings contribute to the advancement of the social sciences.

Empirical Inference in Life

Social science seeks to understand the behavior of individual human beings and their social interactions. In their day-to-day lives, ordinary people face problems of empirical inference-- problems of both identification and statistical inference--similar to those that confront social scientists. People facing inferential problems are subject to the same rules of logic as are social scientists. The conclusions that people can logically draw are determined by the assumptions and the data that they bring to bear.

Social scientists need to keep this constantly in mind as we seek to model and interpret human behavior. We do not know much about how people deal with the inferential problems that they face. Economists have been particularly negligent. Economists usually suppose that people's empirical inferences are expressed in their expectations for the future. Expectations are a subjective concept, but economists have long exercised a self-imposed prohibition on the use of subjective data in empirical analysis. Rather than seek to learn about expectations, economists have generally made assumptions about expectations.

The rational expectation assumptions commonly made by economists may be elegant and analytically appealing. However, they have little empirical support. In many applications, accepting a rational expectations assumption means accepting the idea that ordinary people somehow are able to solve identification problems that have long challenged social science research. As I see it, ordinary people--like social scientists--have to cope with ambiguity.

3. The Selection Problem

Social scientists constantly ask "treatment effect" questions of the form: What is the effect of ____ on ____? For example, what is the effect of welfare programs on labor supply? What is the effect of schooling on wages? What is the effect of the sentencing of offenders on recidivism?

Empirical analysis of treatment effects poses a fundamental identification problem, which is commonly called the selection problem. The researcher wants to compare the outcomes that people would experience if they were to receive alternative treatments. However, treatments are mutually exclusive. At most, the researcher can observe the outcome that each person experiences under the treatment that this person actually receives. The researcher cannot observe the outcomes that people would have experienced under other treatments. These other outcomes are counterfactual. Hence, data on treatments and outcomes cannot by themselves reveal treatment effects.

The Returns to Schooling

Ordinary people want to learn treatment effects in everyday life and so face the selection problem. Consider, for example, young people deciding whether to continue their schooling or to enter the labor market. To make good decisions, young people want to learn their returns of schooling. They may be able to observe the outcomes experienced by family, friends, and others who have made their own past schooling decisions. However, they logically cannot observe what outcomes these people would have experienced had they made other decisions. Thus, young people making schooling decisions in ordinary life are "adolescent econometricians" who face identification problems similar to those that have made it so hard for labor economists to agree on the returns of schooling (Manski 1993b).

Random Treatment Selection

Point identification of treatment response requires assumptions about the process of determining treatment selection and outcomes. The most longstanding practice, and still the most prevalent one, is to assume that, among people with specified observable covariates, treatment selection is statistically independent of outcomes. This assumption is variously called random, exogenous, or ignorable treatment selection. The specified covariates are often, misleadingly, said to "control for" treatment assignment.

The assumption of random treatment selection is appropriate in the analysis of data from classical randomized experiments. Indeed, this is the reason why randomized experiments are valued so highly. The assumption of random treatment selection is usually suspect in non-experimental settings, where observed treatments may be self-selected or otherwise chosen purposefully. Over the years, a variety of alternative assumptions have been proposed and applied to non-experimental data. Indeed, the development by econometricians of latent-variable models and instrumental-variable approaches in the 1970s was initially greeted with enthusiasm as "solving" the problem of identifying treatment effects from non-experimental data. It soon became apparent, however, that these approaches replace the suspect assumption of random treatment selection with alternative assumptions that are no less suspect.

Comparing Treatments Using the Empirical Evidence Alone

My research has moved away from the conventional focus on assumptions that yield point identification of treatment response. I began by asking what can be learned about treatment response from the empirical evidence alone, given no assumptions about the process generating treatments and outcomes (Manski 1990). I found that this question has a simple answer. The observation of realized treatments and outcomes does imply restrictions on the distributions of outcomes under alternative treatments. However, the data are necessarily consistent with the hypothesis that there is a common distribution of outcomes under every treatment. Hence, empirical evidence alone cannot determine whether one treatment is better than another.

Consider, for example, the question of how judges should sentence convicted offenders. The two treatments might be imprisonment and probation. The outcome of interest might be recidivism: Does the offender commit a subsequent crime? In this setting, the objective might be to learn the classical treatment effect: the difference between the recidivism rate that would occur if all offenders were imprisoned and that which would occur if all offenders were sentenced to probation.

Suppose first that one has no data at all. Then all one can say about the classical treatment effect is that it lies between --1 and 1, an interval of width 2. The treatment effect is --1 if the probability of recidivism is 0 under the mandatory imprisonment policy and is i under the mandatory probation policy. The effect is 1 if these policies have the opposite consequence.

It can be shown that observation of realized treatments and outcomes enables one to cut the width of this interval exactly in half, to an interval of width i rather than 2. Thus, data alone solve half of the identification problem. Assumptions are needed to solve the other half.

The location of the interval within which the treatment effect must lie depends, in a simple way, on the empirical pattern of the data on treatments and outcomes. Whatever the interval is, it necessarily contains the value 0. Hence, the data alone do not suffice to determine the sign of the treatment effect.

An Empirical Illustration

Manski and Nagin (1998) analyzed observational data on the sentencing of 13,197 juvenile offenders in the state of the and their subsequent recidivism. We compared recidivism under the two main sentencing options available to judges: confinement in residential facilities (t = 1) and sentences that do not involve confinement (t = 0).

Let the outcome take the value y = I if an offender is convicted of a subsequent crime in the 2-year period following sentencing, and y = 0 otherwise. The empirical distribution of treatments and outcomes among the observed offenders was found to be as follows:

probability of residential treatment = 0.11,

probability of recidivism conditional on residential confinement = 0.77,

probability of recidivism conditional on nonresidential treatment = 0.59.

The problem is to use this empirical evidence to draw conclusions about the probabilities of recidivism under two hypothetical policies--one in which all offenders are confined and the other in which none is. Someone willing to assume that judges presently sentence offenders randomly can conclude that these recidivism probabilities are 0.77 and 0.59, respectively. However, the empirical evidence reveals only that they fall in the ranges [0.08, 0.97] and [0.53, 0.64].

Random sentencing does not seem a particularly credible assumption, so there is a clear need for empirical research analyzing how judges actually make sentencing decisions. The criminology and sociology literatures report some descriptive, usually qualitative, studies of judicial behavior. However, these studies do not yield much information on judges' decision processes.

A critical open question concerns the way that judges make their own inferences about recidivism. Judges need to know treatment response if they are to make effective sentencing decisions. Thus, in their everyday professional lives, judges confront the same identification problem as do criminologists who study sentencing and recidivism.

Treatment Choice Under Ambiguity

Recently, I have begun to explore the implications of the selection problem and other identification problems for treatment choice (Manski 2000, 2002). I suppose that a social planner must choose a treatment rule that assigns a treatment to each member of a heterogeneous population. The planner could, for example, be a physician choosing medical treatments for each member of a population of patients, a school official making course-placement decisions for each member of a population of students, or a judge deciding sentences for each member of a population of convicted offenders.

I suppose that the planner observes certain covariates for each person. These covariates determine the set of treatment rules that are feasible to implement. The set of feasible rules is the set of all functions mapping the observed covariates into treatments. Each member of the population has a response function mapping treatments into real-valued outcomes. I suppose that the planner wants to choose a treatment rule that maximizes a utilitarian social welfare function.

Suppose, for simplicity, that all treatments have the same costs. Then it is easy to show that an optimal treatment rule assigns to each member of the population a treatment that maximizes mean outcome conditional on the person's observed covariates. The planner faces a problem of treatment choice under uncertainty if he knows the conditional mean responses and, consequently, can implement an optimal rule. The planner faces a problem of treatment choice under ambiguity if he does not know enough about mean response to be able to implement an optimal rule. The term ambiguity dates back at least to Ellsberg (1961). Economists sometimes refer to ambiguity as Knightian uncertainty.

The general presumption among economists has been that economic agents face problems of choice under uncertainty. I argue that identification problems make ambiguity a fundamental problem of treatment choice in practice. Although empirical evidence on realized treatments and outcomes does imply informative bounds on mean responses under alternative treatments, these bounds necessarily overlap. Hence, observations of realized treatments and outcomes do not suffice to rank the feasible treatment rules.

The fact that the empirical evidence does not enable the determination of the optimal treatment rule does not imply that a planner should be paralyzed, unwilling and unable to choose a rule. However, it does imply that the planner cannot assert optimality for whatever rule he does choose. A planner who acts as a subjective Bayesian or who uses the maxi-min rule may sensibly assert that he is using a "reasonable" decision rule, but he should not assert that he is using an optimal rule.

Ambiguity in Everyday Life

Ordinary people face ambiguity as do social planners. Manski (2003a) analyzes social interactions that stem from the successive endeavors of new cohorts of heterogeneous decision makers to learn from the experiences of past cohorts. A dynamic process of information accumulation and decision making occurs as the members of each cohort observe the experiences of earlier ones and then make choices that yield experiences observable by future cohorts. Decision makers face the selection problem as they seek to learn from observation of past actions and outcomes while not observing the counterfactual outcomes that would have occurred had other actions been chosen. Under the assumption that all cohorts face the same outcome distributions, I show that social learning is a process of sequential reduction in ambiguity. The specific nature of this process, and its terminal state, depend critically on how decision makers make choices under ambiguity. I use the problem of learning about innovations to illustrate.

4. Identification of Market Demand

The Law of Decreasing Credibility

Determining what can be learned using the data alone provides a logical starting point for empirical analysis but ordinarily will not be the ending point. Having determined what can be learned in the absence of assumptions, we should then ask what more can be learned if assumptions of different strengths and degrees of plausibility are imposed. As a methodologist, I have not advocated that empirical researchers or social planners make one particular assumption or another. My objective has been to provide a menu of possibilities. I have particularly wanted to clarify the dilemma that researchers and planners face as they decide what assumptions to maintain. I have called this dilemma (Manski 2003b) the Law of Decreasing Credibility. The credibility of inference decreases with the strength of the assumptions maintained.

Classical Econometric Analysis of Demand

To demonstrate the Law of Decreasing Credibility, consider the most venerable of all identification problems in econometrics: the identification of market demand from observations of market equilibriums. Economic analyses of market demand usually suppose that there is a set of isolated markets for a given product. Each market is characterized by a demand function, which gives the quantity of product that price-taking consumers would purchase if the price were set at any specified level. In each market, the interaction of consumers and firms determines the price at which transactions actually take place.

The classical econometric analysis of market demand achieves identification through two critical assumptions. One is that demand varies linearly with price, with the same slope in every market. The other is that demand is mean-independent of an observed covariate, termed an instrumental variable.

Consider these assumptions. Economic theory does not suggest that demand should be linear in price or, indeed, that demand should be any particular function of price. All that economic theory does suggest is that demand should be downward sloping in price. Of course, even this is not a universal prediction; for example, texts regularly note the possibility of Giffen goods. However, the ordinary presumption of economists is that demand is downward sloping in price. It certainly is not that demand is linear in price.

Economic theory does not suggest that demand should be mean-independent of any particular covariate, so the credibility of this assumption must be assessed on a case-by-case basis. The assumption often is suspect in practice. Empirical researchers regularly debate whether some covariate is or is not a "valid instrument."

Monotone Treatment Response

Concerned that classical econometric analysis is built on fragile foundations, I have, in a series of papers, studied what may be learned about market demand when weaker, more credible assumptions are imposed.

Manski (1995, 1997) investigated what may be learned when one assumes only that demand is downward sloping in each market. My analysis assumed nothing else about the shape of demand functions and, moreover, assumed nothing about the process of price determination. The basic idea is that observation of an equilibrium (quantity, price) pair implies that some downward sloping demand function passes through this point. Aggregating across markets yield bounds on the distribution of demand functions. These bounds reveal what basic economic theory implies for the econometric analysis of market demand.

In the language of the analysis of treatment response, downward sloping demand is an assumption of monotone treatment response. In this language, prices are treatments, quantity demanded is an outcome, and the demand function is a response function mapping treatments into outcomes. So the econometric analysis of market demand is an instance of the analysis of treatment response.

The assumption of monotone treatment response is credible in many applications other than analysis of market demand. Another important economic application is to production analysis. There, inputs are the treatments, outputs are outcomes, and the production function is the response function. In this case, the assumption that response functions are monotone means simply that output weakly increases with inputs. An additional credible assumption in production analysis may be that response functions are concave, so that increases in inputs have diminishing marginal returns. Manski (1997) studied the identifying power of this assumption, as well as that of monotonicity.

Using Instrumental Variables

Just as it may enhance credibility to replace the assumption of linear demand with one of downward sloping demand, it may also enhance credibility to use instrumental variables without making accompanying assumptions about the form of treatment response. Manski (1990, 1994) showed that, if outcomes are bounded variables, the mean-independence assumption commonly assumed in demand analysis and elsewhere generally does not point-identity treatment response but does yield informative bounds on mean treatment response. Manski (2003b) presents stronger findings when one imposes a statistical-independence assumption rather than a mean-independence assumption. These bounds are easy to implement. They are intersections, across different values of the instrumental variable, of the bounds that apply when the empirical evidence is used alone. Thus, an empirical researcher studying market demand can use an instrumental variable without also having to restrict how demand varies with price.

Manski and Pepper (2000) studied the identifying power of monotone instrumental variables. These weaken the equality defining a traditional mean-independence assumption to a weak inequality, yielding a new assumption that may be credible when mean independence is not.

Consider, for example, the continuing effort by labor economists to learn the returns to schooling. Many empirical articles estimate regressions of wages on schooling and interpret the findings as estimates of the returns to schooling. As is well known, this interpretation is appropriate only if schooling is exogenous, in the sense that wage functions are mean-independent of chosen levels of schooling. However, many economic models of schooling choice and wage determination predict that persons with higher ability have higher mean wage functions and tend to choose higher levels of schooling than do persons with lower ability. So the assumption of exogenous schooling is suspect. However, it may be credible to assume that people who select higher levels of schooling have weakly higher mean wage functions than do those who select lower levels. This assumption makes level of schooling a monotone instrumental variable.

5. Other Identification Problems

I have dwelled on the selection problem--that is, the unobservability of counterfactual outcomes. This is an especially important and distinctive identification problem. However, it is hardly the only one confronted in social science and in everyday life.

Mundane problems of missing outcome and covariate data arise regularly in empirical research. It has been common to "solve" the problem by assuming that data are missing at random, but this assumption is rarely justifiable. As when confronting counterfactual outcomes, it is important to know what can be learned from the available empirical evidence alone. Aspects of this problem have been studied in Manski (1989, 1994), Horowitz and Manski (1998, 2000, 2001), and Zaffalon (2002). The related problem of missing treatment data in the analysis of treatment response has been studied by Molinari (2002a). Another related problem is inference with interval-measured data (Manski and Tamer 2002; Haile and Tamer 2003).

Missing data, as problematic as they may be, pose less severe an identification problem than do contaminated data. In contaminated sampling problems, the observable data are a mixture of "good" observations drawn from the distribution of interest and "bad" observations drawn from some other distribution. The researcher does not know which data are good and which are bad. Aspects of this problem have been studied in Horowitz and Manski (1995), Bollinger (1996), Hotz, Mullin, and Sanders (1997), Cross and Manski (2002), Dominitz and Sherman (2002), Kreider and Pepper (2002), and Molinari (2002b).

Identification problems pervade every aspect of empirical research and every attempt by ordinary people to learn about the world in which they live. Overall, I view myself as presenting a mixed message. My pessimistic side would argue that it is rarely possible to credibly "solve" identification problems. The optimist in me would argue that the more we understand these problems, the better we will be able to cope with them.

References

Bollinger, Christopher. 1996. Bounding mean regressions when a binary variable is mismeasured. Journal of Econometrics 73:387-99.

Cross, Philip, and Charles Manski. 2002. Regressions, short and long. Econometrica 70:357-68.

Dominitz, J., and R. Sherman. 2002. Nonparametric analysis of mixture models with verification, with an application to test score data. Unpublished paper, Carnegie Mellon University.

Ellsberg, D. 1961. Risk, ambiguity, and the Savage axioms. Quarterly Journal of Economics 75:643-69.

Haile, Philip, and Elie Tamer. 2003. Inference with an incomplete model of English auctions. Journal of Political Economy. 111:1-51.

Horowitz, Joel, and Charles Manski. 1995. Identification and robustness with contaminated and corrupted data. Econometrica 63:281-302.

Horowitz, Joel, and Charles Manski. 1998. Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and imputations. Journal of Econometrics 84:37-58.

Horowitz, Joel, and Charles Manski. 2000. Nonparametric analysis of randomized experiments with missing covariate and Outcome data. Journal of the American Statistical Association 95:77-84.

Horowitz, Joel, and Charles Manski. 2001. Imprecise identification from incomplete data. Proceedings of the 2nd International Symposium on Imprecise Probabilities and Their Applications. Accessed 13 December 2002. Available http://ippserv.rug.ac.be/~isipta0l/proceedings/index.html.

Hotz, V. Joseph, Charles Mullin, and Seth Sanders. 1997. Bounding causal effects using data from a contaminated natural experiment: Analyzing the effects of teenage childbearing. Review of Economic Studies 64:575-603.

Koopmans, Tjallings. 1949. Identification problems in economic model construction. Econometrica 17:125-44.

Kreider, Brent, and John Pepper. 2002. Disability and employment: Reevaluating the evidence in light of misreporting errors. Unpublished paper, University of Virginia.

Manski, Charles. 1989. Anatomy of the selection problem. Journal of Human Resources 24:343-60.

Manski, Charles. 1990. Nonpammetric bounds on treatment effects, American Economic Review Papers and Proceedings 80:319-23.

Manski, Charles. 1993a. Identification of endogenous social effects: The reflection problem. Review of Economic Studies 60:531-42.

Manski, Charles. 1993b. Adolescent econometricians: How do youth infer the returns to schooling? In Studies of supply and demand in higher education, edited by Charles Clotfelter and Michael Rothschild. Chicago: University of Chicago Press, pp. 43-57.

Manski, Charles. 1994. The selection problem. In Advances in econometrics, Sixth World Congress, edited by Christopher Sims. Cambridge, UK: Cambridge University Press, pp. 143-70.

Manski, Charles. 1995. Identification problems in the social sciences. Cambridge. MA: Harvard University Press.

Manski, Charles. 1997. Monotone treatment response. Econometrica 65:13 11-34.

Manski, Charles. 2000. Identification problems and decisions under ambiguity: Empirical analysis of treatment response and normative analysis of treatment choice. Journal of Econometrics 95:415-42.

Manski, Charles. 2002. Treatment choice under ambiguity induced by inferential problems. Journal of Statistical Planning and Inference 105:67-82.

Manski, Charles. 2003a. Social learning from private experiences: The dynamics of the selection problem. Review of Economic Studies. In press.

Manski, Charles. 2003b. Partial identification of probability distributions. New York: Springer-Verlag.

Manski, Charles, and Daniel Nagin. 1998. Bounding disagreements about treatment effects: A case study of sentencing and recidivism. Sociological Methodology 28:99-137.

Manski, Charles, and John Pepper. 2000. Monotone instrumental variables: With an application to the returns to schooling. Econometrica 68:997-1010.

Manski, Charles, and Elie Tamer. 2002. Inference on regressions with interval data on a regressor or outcome. Econometrica 70:519-46.

Molinari, Francesca. 2002a. Missing treatments. Unpublished paper, Northwestern University.

Molinari, Francesca. 2002b. Identification of probability distributions with misclassified data. Unpublished paper, Northwestern University.

Zaffalon, Marco. 2002. Exact credal treatment of missing data. Journal of Statistical Planning and Inference 105:105-22.

Charles F. Manski *

* Department of Economics and Institute for Policy Research, Northwestern University, 2003 Sheridan Road, Evanston, IL 60208: E-mail [e-mail].

This paper is based on my Association Lecture delivered to the Southern Economic Association at its 2002 meeting in New Orleans.