The theory of change, as well as the economic method selected, will also affect how that method is applied. The cost analysis must clearly correspond to the scale, scope, duration, and intensity of the intervention that is being implemented. For instance, we can think of a high school support program such as Talent Search that is intended to boost high school completion and college enrollment (Bowden & Belfield, 2015). If the mechanism by which Talent Search is effective is that students successfully apply for college, then the analyst should cost out that application process. Alternatively, if the mechanism is continuous mentoring support through high school, then the costing exercise will have to follow students through their mentoring experiences, perhaps over multiple years. In turn, this means that the comparison group will have to be followed over the same time period. As another example, we can think of socioemotional learning interventions that are intended to change the climate across the school. The cost analysis must measure all changes in resources at the school level (see Long, Brown, Jones, Aber, & Yates, 2015). Cost per student may not be an accurate guide: In order to influence school climate, all (or most) of the students must receive services collectively, not individually, and it is the total cost that is salient.
Similarly, the economic analysis must be responsive to the context in which the intervention is delivered. On the cost side, the availability and prices of ingredients may vary across school districts: In low-income countries, some inputs (e.g., online computing systems) may not be available; in areas of high poverty, the analyst must take into account the opportunity cost of time for schoolchildren who may need to work to support their families. Context also matters on the benefit side. For example, a preschool program might reduce future youth crime—in fact, this effect is one of the main reasons why the HighScope Perry Preschool Program has such a high payoff. But this impact may only be salient for preschools in localities where the crime rate is high (a distinction noted in Barnett & Masse, 2007).
Finally, the analyst may have to infer longer-term outcomes from intermediate ones. In the case of preschool, for example, the researcher cannot wait 15 years for information on earnings; even freshman college programs can have very delayed impacts on future earnings. In these cases, it is important that the BC analyst choose to measure intermediate outcomes (e.g., high school graduation or first-semester credits) that are well correlated with longer-term outcomes. It is also important that any changes in behavior be measured when they occur and for how long they occur. Interventions where change is “fast-acting” and persistent are of higher value because of their immediacy, and the BC analysis should reflect this.
At this stage, we can cover only the main issues. As the theory of change for each intervention becomes more detailed, the economic evaluator should respond accordingly. Fundamentally, each economic analysis should adapt to fit with the theory of change for each specific intervention.
2.4. Determining if Economic Evaluation Is Necessary
As a final consideration before beginning the research, it is worth asking whether an economic analysis is needed. Thus far, it has been assumed that in most cases cost analysis should be done and that a lot of valuable information will be obtained from such analysis. However, this reasoning is not definitive. We have already suggested that if there are no alternatives or if a decision is already predetermined, the entire evaluative situation—including economic analysis—becomes moot. Further, if sufficient time or other resources are lacking or if CE types of data will not alter decisions, there is probably not a strong case for doing cost analysis. Even when all the prerequisites for implementing and using cost analysis are present, it is important that such an evaluation be worth doing in the first place.
Moreover, economic evaluation techniques might themselves be interrogated using efficiency criteria: What is the economic payoff to performing CE analysis? Do the benefits of performing BC analysis outweigh the costs? In response, we refer to Scriven’s (1974, pp. 85–93) notion of cost-free evaluation. In considering the choice of an analytic approach and supportive design for implementation, one must ask what will be gained by finding a better alternative. If the gains will be relatively small, only a small investment in evaluation would be merited. Indeed, if the potential gains of a good decision are minuscule, it is possible that no formal analytic study should be undertaken. That is, intuition and present knowledge should suffice, given that little will be gained from a more formal and extended evaluation. However, when the value of selecting a new alternative can be very great, it may be worth making a large investment in evaluation and analysis. Scriven would say that this situation meets the cost-free evaluation criterion if the probable gains from the evaluation will be in excess of the costs of the evaluation. It is cost-free in that more is saved by a good and appropriate evaluation than is expended in resources on that evaluation. Put simply, the more costly the intervention is to implement, the more likely it is that an economic evaluation will pay for itself by changing the allocation of resources. Prima facie, the economic case for economic evaluation appears reasonable: Society devotes a substantial amount of resources to education programs, and the information obtained from an economic evaluation is probably substantial and salient.
Nevertheless, in order to know how much economic evaluation to do, one needs a sense of what research effort is involved. We describe the research tasks in detail in the following chapters, so here we offer very general comments. First, it is important to recognize that costs are estimated, just in the same way that impacts are estimated. Cost estimation is therefore a substantial research activity, just as identification of impacts is. When estimating impacts, the researcher is immediately concerned with potential sampling error—that is, the likelihood that the sample chosen reflects the population. Sampling error is ameliorated by properly designing the sampling frame and drawing a sample of sufficient power. This logic also holds when estimating costs: The researcher should draw from the population of sites (e.g., schools or classrooms or students) to ensure that the sample reflects the population. A priori, it is difficult to say if the sample for estimating costs should be larger or smaller than the sample for estimating impacts; it depends on the variation of costs within the population. Overall, whatever research tasks are important for impact evaluations are likely to be required of economic evaluators.
Second, in most BC or CE evaluations, some expertise will be needed from persons who are specialists in these kinds of analyses. Although BC analysis is a standard component of the economics of public finance, many academic economists have not had practical training in either cost analysis or shadow pricing techniques. Also, one should not confuse expertise in accounting with expertise in BC or CE analysis. While BC and CE analysts must have a good understanding of cost accounting, cost accountants need have no understanding of BC and CE analysis. Often, the correct approaches for cost accounting for business firms are inappropriate for estimating the costs of social projects. In fact, one of the rationales of public economics is that, frequently, social costs and benefits differ considerably from private ones. Cost accountants are not usually trained to address social costs and benefits.
That said, economic evaluations do not just rely on economic principles. In order to perform CE analysis, one has to determine what effectiveness means in each educational context. For instance, for reading programs it might be comprehension or vocabulary or fluency; for dropout prevention programs it might be keeping students in school each year, getting them to finish 12 grades, or getting them to complete a high school diploma. The answers to these questions cannot be found in economic theory. Experts in the respective fields are needed to determine which specific outcomes are most valid.
Overall, there is no presumption that an economic evaluation is necessary nor is there a fixed amount of effort that an expert might contribute to perform such an evaluation. Resources would