Evidence-based Policy Making: Its Significance and Role.
What Is EBPM?
Analyses based on social sciences such as economics have thus far tended to not be emphasized in the policy making process in Japan. Instead of choosing the optimal policy based on estimations of effects and costs, policy making was sometimes carried out based on the instinct of the policy planners, or in response to the demands of the stakeholders.
Unfortunately, there have been several examples recently where policies were determined without the literacy regarding basic social sciences.
The first example is the issue of labor hours data around the discretionary working system. In February 2018, a data deficiency issue emerged in relation to the Work Style Reform Bill, a bill to strengthen regulations on labor hours, expand the target workforce for the discretionary employment system, and other matters. Data sets that were introduced claimed that labor hours for those workers under the discretionary working system were shorter than those of ordinary workers, but it was later revealed that it was impossible to compare the two data sets. Consequently, sections of the discretionary employment system were removed from the bill.
The second example is the experiment on "Cacao Rejuvenating the Brain" which was conducted by the "Impulsing Paradigm Change through Disruptive Technologies Program" (ImPACT) of the Cabinet Office. This experiment sought to examine whether chocolates with greater amounts of cacao can increase the amount of cerebral cortex, and its result that "cacao is likely to increase the amount of cerebral cortex and elevate learning functions" was released (January 2017) based on data from just 30 samples from people who ate chocolate. Normally, data comparison between those who ate chocolate and those who did not was required, but it was never conducted. As an experiment, it had fatal errors.
These errors would have been avoided (or at least discovered sooner) had there been basic literacy concerning data analysis.
Moreover, there are more positive reasons why we should introduce more social science analyses into policy making. With a declining birth rate and aging population and the current financial stringency, there is a need to use policy resources as effectively as possible. Policies that may appear to be instinctively effective can, in reality, cause great side-effects. There is greater need to utilize social sciences expertise in the policy making process.
The concept of "evidence-based policy-making" (EBPM) becomes important in this respect. This concept requires accurate data and robust methodology to analyze the effects and costs of policy options and present these as evidence when determining a policy.
Interests and Ideas
Let us examine the significance of EBPM from a political science perspective that deals with factors influencing policy making. The two factors that are important in influencing policy making are the two "i"s: interest and idea.
Interest refers to the self-interest held by each actor, or more concretely, interests such as wealth and power. In many political science theories, political actors are thought to act on the objective to achieve self-interest. Pluralism, which was once the dominant theory in American political science, is a theory that looks at politics with the activities of interest groups at its core. Rational choice theory, which has been booming in recent years, assumes that actors take action to maximize self-interest. For example, politicians are assumed to aim for reelection or promotion (such as ministerial appointments), and government officials are thought to aim for expansion of the budget or the organization, or promotion within the government.
On the other hand, political actors often have ideational goals which are not necessarily tied to material self-interest. Politicians and government officials certainly do have interest in their own reelection or promotion, but on top of that they are generally keen to solve various issues and challenges that occur within their society and act to achieve ideas that make the society a better place. In other words, idea offers objective and method in achieving public interest.
According to political scientists Judith Goldstein and Robert O. Keohane who have been focusing on the role of ideas, ideas can be classified into three categories (Judith Goldstein and Robert O. Keohane, "Ideas and Foreign Policy: An Analytical Framework", in Judith Goldstein and Robert O. Keohane (eds.), Ideas and Foreign Policy: Beliefs, Institutions, and Political Change, Ithaca, NY: Cornell University Press, 1993). First, there is the world view which is the most comprehensive level of category. Religious belief represents this category. The second is principled belief. It represents ideas that show what is right and wrong. Rules such as "human rights should be respected" fit this idea. The third is causal belief, or ideas on the cause and effect relationship. A typical example is the knowledge of natural sciences and social sciences, such as the theory that atmospheric carbon dioxide increases lead to global warming.
What is relevant for EBPM is the third causal belief. EPBM mainly focuses on analyzing what effect a certain policy intervention causes based on scientific evidence.
But so far, policy making has tended to emphasize the private interests of powerful interest groups over public interest, and analysis of causal relations based on scientific evidence was taken rather lightly. In other words, the main factors behind many policy makings have been interests (private interests), and even when the policy making was based on ideas, they lacked precise scientific evidence (in these cases, "ideas" merely refer to improvised thoughts).
In this sense, the aim of EBPM is to reclaim the original definition of "policy making based on ideas", or, policy making based on ideas backed by proper evidence.
EBPM in the UK and US
EBPM has been implemented primarily in the United Kingdom and United States.
In the UK, there are government officials called Government Economists in the Civil Service and they constitute a group called the Government Economic Service. The work conducted by Government Economists includes analyses based on evidence for policy making, monitoring and evaluation of policies that have been implemented, and prioritization of policies from the perspective of economic impact and efficiency.
Government Economists are appointed in each ministry and play a certain role in the policy making of the ministries. For example, the Government Economists draw up the cost-benefit analysis statements attached to the budget proposals for each of the ministries. The total number of Government Economists in the UK has sharply increased in the 2000s. When Prime Minister Tony Blair took office in 1997, there were a total of 515 people, and the number grew to 812 in 2003, 1,200 in 2005, and 1,350 in 2006 (Chart). The Blair government advocated EBPM and brought a sharp increase in the number of Government Economists. Hence political leadership was crucial in the establishment of EBPM in the UK.
In addition to these Government Economists, groups of government officials that engage in EBPM also include Government Social Researchers who are experts in social sciences such as sociology and political science, and Government Statisticians who are statistics experts. The characteristic of the UK government is that these specialized government officials support EBPM.
The experts within the government have been described thus far, but EBPM has also progressed in government-related organizations as well in the UK. The first such organizations that need to be introduced are the What Works Centres (WWCs). WWCs were established under the administration of Prime Minister David Cameron to promote efficient development of effective policies. More concretely, they aimed to create evidence (support and review of primary research), communication (devise to use evidence more readily on site), and application (present guidelines based on evidence). Currently there are WWCs established in various fields such as medical care and health care, education, crime prevention, employment, and aging society measures.
The Behavioral Insights Team (BIT) also utilizes the knowledge of behavioral economics to support EBPM. BIT was initially established as an organization within the Cabinet Office, and transformed to become an independent organization (Social Purpose Company) in 2005. BIT aims to establish evidence mainly through experiments (such as Randomized Controlled Trial, to be discussed later). For example, famous experiments that were conducted in its early days looked at improving the tax collection rate. This experiment has proven that just by sending out a letter that says "people around you all pay taxes" improved the tax collection rate. This was based on the knowledge of behavioral economics which says that appealing to people on social norms can change their behavior.
In the US, EBPM greatly progressed under the administration of President Barack Obama. The Office of Management and Budget (OMB) was at the core of such efforts. The OMB had been notifying all government ministries and agencies of the guidelines for budgeting which were based on EBPM. The content of the guidelines included drawing up budget demands based on solid evidence, making proposals to further enhance evidence, and assigning budgets based on the level of commitment to evidence.
Knowledge of behavioral economics is also utilized in the US as is done in the UK. The Social Behavioral Sciences Team which was set up at the White House is an organization tasked to implement improvements by applying knowledge of behavioral economics. For example, it has conducted experiments on prevention of delinquent public payments and improvements in enrollment in the retirement pension system (references for government related organizations and EBPM in the US were from the reports given by Yohei Kobayashi on "EBPM by Independent Institutions in the UK", and by Hirokazu Tsuda on "EPBM at US Government" at the Symposium "To Promote Policy Planning based on Evidence" hosted by the Research Institute of Economy, Trade and Industry on Dec. 19, 2017 (https://www.rieti.go.jp/en/events/17121901/info.html).
But since the President Donald Trump took office, there has been an incident where the movement of EBPM seems to have slowed down. In December 2017, there was an order placed for organizations under the Department of Health and Human Services banning the use of certain terminology in their documents regarding next year's budget request. The list of banned terminology given to the Centers for Disease Control and Prevention under the department included "evidence-based" and "science-based" ("Trump administration bans selected words at health agencies", Reuters, Dec. 17, 2017). These policies were naturally met with resistance, but it is important to monitor what position the Trump administration, which seems to calmly ignore scientific knowledge, will take on EBPM and whether EBPM in the US will progress or regress.
Methods of EBPM
There is a need to identify the causal relations between a certain policy intervention and its effect to properly determine the policy impacts. Inference on the relationship between cause and effect is called causal inference. Some of the major methods of causal inference are illustrated below (The Economics of Cause and Effect - Thought Process that Catches Truth from Data, Makiko Nakamuro and Yusuke Tsugawa, Diamond, Feb. 16, 2017).
The most ideal is the experimental method. The effect on a group that received policy intervention is compared with a group that did not. The intervened groups and non-intervened groups are assigned randomly and hence this is called the Randomized Controlled Trial (RCT).
This method is considered to be ideal because it is able to eliminate impacts of factors that may interfere with the outcome (It is termed professionally as control). Let us assume the following situation: examining the impact of policy x (for example, employment policies) on an outcome y (for example, decrease in unemployment rate). The sample for this research is two cities A and B. Employment policy x was implemented in city A and it was not in city B. The unemployment rate improved by 1% a year in city A, but there were no improvements in city B. It may thus simply look like policy x created a 1% improvement in employment. But in reality, there may have been other factors that existed in city A but not in city B (for example, products manufactured in city A may have suddenly become famous and by mere chance rise in sales). If that was the case, it cannot be inferred that an improvement in employment is the outcome of x.
If it were an experimental method, the impact of other factors like this can be eliminated, and the sole impact of the intervention which was intended can be measured. For example, the population of a city can be randomly divided into two groups and policy x can be implemented for one group and not for the other (in this instance, the group that received the intervention is called the intervention group and the group that didn't is called the comparison group). By measuring the amount of change in effect y at both groups after a certain period, the amount of effect y policy x has brought can be identified.
While this experimental method is ideal, it requires great cost (including moral issues) in conducting the experiment, and it is not always feasible. When experimental data is not available, statistical methods using existing observational data are conducted. Causal inference using statistical methods includes the following.
First, the natural experiment is very close to RCT. This uses a situation similar to the RCT which developed naturally. For example, for two similar groups of cities, policy x is implemented for one city group, and not for the other group. By using this situation, causal inference can be conducted very similarly to the experimental method (Here, it is important not to leave it to the city to request policy intervention. For example, cities that aggressively seek employment policies may likely be more aggressive in improving employment on other fronts as well. In such cases, even if improvement in employment is observed, it does not necessarily indicate that this is the result of policy x. This is called the endogeneity problem).
Next, the quasi-experiment method conducts causal inference that is close to the experimental methods by devising the usage of observational data. One such method is the "difference in difference" analysis. It is called as such because the difference between before and after the period when the intervention took place for the intervention group is compared with that for the comparison group. This method is valid because it can eliminate the effect of trends. For example, suppose that when an employment policy x is implemented during a particular period, the unemployment rate improves by 1% in a particular city. But it cannot be inferred just by looking at this data that the employment improvement was the result of x. This is because the unemployment rate may have gone down by 1% due to an overall improvement in the economic situation all across the country.
By comparing the level of improvement in the unemployment rate within a group of cities where x was implemented (1% in the above example) and the level of improvement in unemployment within a group of cities where it was not (say, 0.3%), the difference between the two groups (0.7%) can be deduced to be the effect of x.
Other methods include the Regression Discontinuity Design (RDD) and the matching method. The RDD looks at differences in the outcome immediately before and after a particular value. For example, suppose there is a group of students who share similar academic skills. Some of them entered a particular university, but others did not make it to that university with a difference of just a few points in their scores. By comparing their income after university graduation, the impact of attending that particular university on income can be measured by controlling the academic skills. The matching method compares the intervention group with a selected sample from the comparison group that is similar to the intervention group, such as selecting a group of cities from a group of cities where x was not implemented but has various similarities with the group of cities where policy x was implemented, and making comparisons.
Future of EBPM in Japan
In Japan also, there has recently been the introduction of an institutional framework that properly utilizes social science experts, such as in economics, in the policy making process.
The final report compiled by the Statistics Reform Promotion Conference which was released in May 2017 states that "there should be a Director of EBPM Promotion (tentative) appointed to each ministry to direct programs for promotion of EBPM", displaying a policy initiative that fully introduces EBPM. The "2017 Basic Policies for Economic and Finance Management and Reform" from June 2017 states that "In fiscal 2018...programs for both expenditures and revenues should be implemented. Policy making based on evidence should be promoted...in such instances." Along with this governmental policy, each ministry is expected to promote EBPM programs. There are regional governments interested in EBPM as well (such as Hiroshima Prefecture, Saitama Prefecture and Yokohama city).
There are various challenges to implementing EBPM, but the most important challenge is how to obtain specialist-type human resources whose social science analyses can be utilized for policy making. It is currently rather hard to find large numbers of people who can do both the real work of policy making and also conduct social science analysis in Japan. Thus, there is a need to utilize experts such as professors at universities and experts at think-tanks, and also work to establish a system where human resources can receive the necessary training in collaboration with the universities.
On the other hand, there is still a demand for generalist-type government officials to acquire a minimum level of social science literacy. Such literacy naturally includes basic general knowledge such as reading off statistics, but also the aforementioned methods of various causal inferences.
Lastly, I would like to stress that scientific knowledge does not cure everything. Scientific analysis can be useful in better policy making, but one must not assume that it leads to a single "right answer". Appropriate policy making requires listening carefully to expert advice, communicating with public opinion, and making comprehensive decisions. In other words, the role of generalists who have the task of making comprehensive decisions is still important, as is the role of policy analysis specialists. However, it goes without saying that it is better for these generalists to also possess social science literacy.
I hope that EBPM programs in Japan contribute to making a better society.
|Printer friendly Cite/link Email Feedback|
|Publication:||Japan Economic Foundation (Tokyo, Japan)|
|Date:||Jun 30, 2018|
|Previous Article:||Roundtable on Social Innovation in an Aging Society Through Government and Business Collaboration.|
|Next Article:||Roundtable on the Significance of the Scenario Approach in an "Age of Anxiety".|