All You Need to Know About Impact Evaluations

I just read Rosa Muraguri-Mwololo’s excellent and engaged review of the new book Impact Evaluation in Practice, which prompts me in also presenting my take on this important book for the evaluation field, as well as for African countries. The four authors – Gertler, P. J.; Martinez, S.; Premand, P.; Rawlings, L. B.; Vermeersch, C. M – discuss in detail the importance of, techniques for and the methodology of undertaking impact evaluation. You can read the book published by the World Bank here.

The hallmark of Managing for Development Results (MfDR), which in many jurisdictions is also called Results-Based Management (RBM), is the global shift of development focus from inputs, activities and outputs to outcomes and impacts. Outcomes and impacts (also called final outcomes in this book) are referred to as development results since they represent the ‘real’ change in the targeted individuals occasioned by development projects, programmes or policy. The focus on results is increasingly being used to set and track national and international development targets. Results are also being used by, and required of, programme managers to enhance social accountability, inform budget allocations and guide policy decisions. This can only be effectively done if adequate information is available to policy formulators and decision makers.

Impact Evaluations

Impact evaluation assesses the changes in the well-being of targeted individuals that can be attributed to a particular project, programme or policy. An impact evaluation seeks to identify a causal relationship between the project, programme or policy and the outcomes of interest. Impact evaluations seek to answer the question: What is the impact (or causal effect) or a programme on an outcome of interest? Thus, impact evaluation looks at changes in the outcome that are directly attributable to the programme.

Apart from this causality relationship, impact evaluation can also be used to inform policy decisions. This is possible when impact evaluation is used to test the effectiveness of programme implementation alternatives by answering the question: When a programme can be implemented in several ways, which one will be the most effective? Impact evaluations can also be used to gauge the cost effectiveness of a programme or to compare the relative performance of two or more programme alternatives in reaching a common outcome. The effectiveness established through an impact evaluation is compared to the cost of a programme to obtain the cost-benefit balance of that programme.

For multifaceted programmes, impact evaluation helps to answer the following questions:

• What is the impact on one treatment compared with that of another treatment?
• Is the joint impact of a first and a second treatment larger than the sum of the two individual impacts?
• What is the additional impact of a higher intensity treatment compared to a lower intensity treatment?

Impact evaluations are costly undertakings which require strategic deployment of both technical and financial resources. The authors stress that only innovative, replicable, strategically relevant, untested and influential programmes warrant the enormous resources outlay required to conduct high quality impact evaluations.

Techniques for impact evaluation

The authors acknowledge that impact evaluations can either be retrospective or prospective. Prospective impact evaluations are known to produce stronger and more credible results for three reasons: baseline data can be collected to establish pre-programme measures of targeted outcomes; defining measures of a programme’s success in its planning stage focuses the evaluation and the programme on intended results, and the treatment and comparison groups are identified before the programme in implemented.

The basic tenet of an impact evaluation is to compare the change in the outcome brought about by the programme with the change that would occur without the programme, or counterfactual. A key goal in this respect would be to identify a group of programme participants (the treatment group) and a group of non-participants (the comparison group) that are statistically identical in the absence of the programme. In constructing the comparison group, it is important to avoid the two common but highly risky counterfeit estimates of the counterfactual: (i) before-and-after or pre-post comparisons that compare outcomes to programme participants prior to and subsequent to the introduction of the programme; and (ii) with-and-without comparisons between units which chose to enrol and those which choose not to enrol. This presupposes that the theory of change has been clearly formulated.

The book identifies the main techniques for undertaking such comparison as:

• Randomized selection methods
• Regression discontinuity design
• Difference-in-differences (also called Double Difference in many literature)
• Matching

Randomized selection methods and regression discontinuity design produce estimates of the counterfactual through explicit programme assignment rules that the evaluator knows and understands. On the other hand, difference-in-differences and matching methods are useful in situations where the programme assignment rules are blurred or where the first two techniques are not feasible.

The authors note that in all these techniques, the risk of bias is palpable. However this risk can be reduced by using a combination of methods to increase the robustness of the estimated counterfactual. Examples of combined techniques are matched difference-in-differences (matched DD) and difference-in-differences regression discontinuity design (DD RDD).

According to the authors, it is also imperative to avoid imperfect balance where some intended treatment units may not receive treatment or some unintended comparison units may receive treatment. When such imperfect balance occurs, the instrumental variable approach can be used to fix or clear the bias. The instrumental variable approach is valid only under certain circumstances, and is definitely not a panacea to all imperfect balance biases!

Methodology for impact evaluation

The authors have provided a simplified roadmap for implementing an impact evaluation. The steps include preparing for the evaluation, operationalizing the evaluation design, choosing the sample, collecting data, and producing and disseminating findings. Of all these steps, it is important to note that planning for the evaluation is very important for deciding what to evaluate, setting/reviewing objectives and policy questions, developing hypotheses, theories of change, results chain, and choosing indicators. This is inline with the old adage that organizations which fail to plan actually plan to fail! Communicating the findings of the evaluation is perhaps the most important part of the process. For the evaluation to be used for evidence-based policy formulation and implementation, its findings must be disseminated to relevant stakeholders, including the beneficiaries of the programme.

Some concluding reflections

Whereas impact evaluations can produce reliable estimates of the causal effects of a programme, they do not provide insights into programme implementation. They therefore need to be well aligned with a programme’s implementation and guided with information on how, when and where the programme is being implemented. For this reason, impact evaluations and other forms of evaluations are complements for one another rather than substitutes.

While traditional M&E concentrates on outputs and inputs, impact evaluation has been relegated as an unnecessary burden in development interventions. Many programmes are forgotten soon after closure with no one bothering to follow up a few years later to establish the true impact that the programme has delivered to the targeted beneficiaries. Even valuable lessons which could be drawn from the outcomes of the programmes are never tapped.

To demonstrate effectiveness of aid and any development interventions, it is important for governments and development partners to in-build impact evaluations in the design of their programmes. The spirit of MfDR which requires development initiative to demonstrate results emphasise on the realization that results are not outputs, rather they are outcomes and impacts which can only be seen in the medium-term and the long-term after the intervention is completed. True results, therefore, can only be demonstrated through impact evaluations!

I would love to read your thoughts about impact evaluation and how countries can play a better role at implementing this technique.

James O. Wagala, Kenya

Institutional Reform and Capacity Building Specialist

Public Service Transformation Department

Office of the Prime Minister