Impact Evaluation Technical Course: How to Manage, Design, and Conduct Impact Evaluations - April 9 -19, 2013 - Shanghai, China

CLEAR-SHIPDET is offering a technical course at the Asia-Pacific Finance and Development Center (AFDC). This course introduces impact evaluation as a key instrument for determining project/program effectiveness, informing policy development and improving program designs. It covers commonly used econometric and statistical methods to evaluate the impacts of social and other programs in developing countries. The course includes both randomized and non-randomized methods.  We will also cover other practical aspects, such as managing an impact evaluation team, planning and implementing data collection, and disseminating the results of the evaluation.  

Address of SNAI: 200 Panlong Road, Qingpu District, Shanghai, China 201702, Tel: (86 21) 69768000

Course Module Descriptions

Below is a description of each of the modules of the course.

• Impact Evaluation in the policy and program cycles: This session gives a general overview of how evidence and impact evaluations are implemented within the policy cycle, highlighting both its strengths and weaknesses. In this discussion, we explore the development of theories of change for programs. We also identify which types of evaluation questions and policy decisions best align to impact evaluation.
• The need for a counterfactual: This session provides an introduction to counterfactuals. Impact evaluations seek to answer the questions: What would the situation be with or without the intervention? What is the causal impact of the program (or project or policy) on the outcomes of interest?
• Overview of Impact Evaluation Techniques: This session walks participants through a range of popular impact evaluation techniques, including randomized experiments, difference-in-difference, regression discontinuity, matching methods, regression estimators and systematic review.
• Threats to Validity: This session will review impact evaluations that are very instructive regarding how evaluations should be implemented and what types of “threats” can arise regarding both internal and external validity, and how to address these threats.
• Qualitative and Quantitative Methods: This session reviews key concepts related to these methods, as well as the differences and complementarities between quantitative and qualitative methods when conducting impact evaluations.
• Impact Evaluation Management: This session looks at the kinds of competencies and the background required for a team undertaking impact evaluation. The session also provides an overview of the issues on how to disseminate the results of impact evaluations and the kinds of knowledge products that can be demanded in order to make impact evaluation more accessible to support the use of findings for policy and management.
• Ethics and Politics: This session presents ethical and political considerations that must be taken into account when conducting impact evaluations.
• Regression Analysis: Regression analysis is the key statistical tool used in almost all impact evaluations.  This session provides a refresher for regression analysis and introduces the basic commands in the software for several regression models that are often used in impact evaluation, including linear regression models, probit/logit models, and random/fixed effects models.
• Designing Questionnaires and other Data Collection Instruments: This session will provide an overview of the different types of questionnaires and other data collection instruments, and will provide practical advice and examples in survey processes, which are an essential part of many impact evaluations.
• Sample Size, Sample Design, and the Power Calculations: A key issue in any randomized evaluation is how large of a sample is needed, and how the sampling should be done. The session provides detailed information on how to answer these questions, and explains some trade-offs that the evaluator will face.
• Introduction to Regression Estimators: The session will present the four simplest estimators, as well as the assumptions required regarding the process by which people are selected and participate in impact evaluations to ensure that they yield unbiased estimates of impacts when program participation is not randomized.  The four estimators are: cross-section, before-after, difference-in-differences, and within estimator.
• Implementing Randomized Evaluations: In theory, randomized control trials (RCTs) provide accurate estimates of program impacts. Yet, in practice there are many things that need to be considered to obtain accurate estimates from RCTs.  Also, many things can go wrong with RCTs.  This session provides more detailed advice on how to conduct an RCT and discusses potential solutions to problems that may arise.
• The Difference-in-Differences and Within Estimators: This session will explain the importance of the fixed effect errors structure for before-after and difference-in-difference methods, and discuss other issues relating to these two estimators (given that the assumptions needed for the cross-section estimator to be valid are unlikely to hold).
• Introduction to Matching Methods: This includes Cross-sectional matching estimators; Rosenbaum and Rubin (1983) theorem for addressing curse of dimensionality; implementation of propensity score matching estimator; nearest-neighbor methods; nonparametric methods; and determination of the overlap support region. 
• Data Collection, and Survey and Data Management: This session will focus on data collection and data management issues, including finalizing the questionnaire design, field operations, data entry, and data “cleaning”.