Let’s delve into what it means to “evaluate an intervention” by answering some non-obvious questions: what can be evaluated, how, and also why.
This article marks the beginning of our Guide’s collaboration with ASVAPP, with whom we will explore the topic of evaluation. Article edited by Valentina Battiloro and Luca Mo Costabella.
Evaluation: what and how
How do you design an evaluation? How and when do you carry it out?” are increasingly common questions when designing an intervention, with the aim of solving a given collective problem. In recent decades, the increased attention on the subject of evaluation has been accompanied by benefits in terms of knowledge, but also by some confusion.
To clarify a bit, it is convenient to start with the term. “Evaluate” means to make a judgment: it implies a comparative activity between what is observed as a result of an intervention and a term of comparison, such as a standard level, or expected conditions.
To whom is the judgment addressed? Not to the entity that designs, finances, or implements the intervention. What then should judgment be made about, and why?
The objects of judgment and cognitive needs that move evaluation are potentially many, and different questions imply the use of different analytical approaches. In a schematic and non-exhaustive way, we can identify three different expressions of judgment that are aimed at evaluating different aspects of an intervention, and each moves specific evaluative approaches:
- The reporting of achievements;
- the implementation analysis;
- the impact assessment.
Evaluation, what has been done: the reporting of achievements
One of the most frequent cognitive needs for those who fund and design interventions is to keep track of and communicate to the outside world what has been done with the resources at hand. Accomplishment reporting brings its gaze to the numbers and characteristics of the resources invested, the actions carried out, the recipients of the intervention and their eventual post-project conditions. It is an evaluation activity because it involves a comparison between what was done and what was imagined to be done, or stated to be done.
This form of evaluation involves an ad hoc data collection system, the design of which must necessarily precede the initiation of the intervention. Designing this activity involves some indispensable cautions:
- Identify in a timely manner useful information to keep track of the progress of the intervention and its achievements;
- Choose a unique mode for recording information. Often we are dealing with interventions implemented in multiple local organisations, by multiple implementers, and targeting a multiplicity of beneficiaries: identifying unique ways of recording information allows us to make comparisons and aggregations that are necessary when we want to know about achievements and results;
- Define the baseline measures useful for checking the congruity between what has been achieved and what is expected or desired.
What has changed “because of” the impact assessment
The third cognitive need is for the effectiveness of the intervention, that is, its ability to change the original problem. This is the domain of impact (or effects) evaluation: a type of evaluation that involves observing or measuring the conditions of interest post-intervention, and credibly estimating what would have happened without intervention.
The effect (or impact) is estimated as the difference between the two scenarios, the first of which (factual) exists and is measurable; the second (counterfactual) has to be reconstructed and somewhat estimated. Approaches to counterfactual estimation are diverse (see here for a brief overview), and rely heavily on the use of appropriately selected control groups. This type of estimation requires specific skills (statistical and econometric) and considerable effort.
Certain feasibility conditions are also necessary for its implementation: the availability of adequate data, the possibility of constructing a control group that is truly comparable with the “treated,” and a sufficient size of the groups analyzed to ensure the accuracy of the estimates. The strength of this type of evaluation lies in the possibility of providing answers that are yes circumscribed, but solid and precise, with a much reduced degree of subjectivity compared to any other approach.
While the first two activities accompany the implementation of the intervention (although they may be carried out after it is completed), impact evaluation is necessarily an ex-post activity, which can be initiated only after the intervention is completed or after a useful window of time has elapsed to capture any changes it produces on the original problem.
Gear up before you go: the theory of change and the logical framework
Before asking what one wants to know, however, it is necessary to consider the nature of the intervention being evaluated. This involves taking a step back from the requirements that drive the evaluation, to reconstruct the “theory of change” of the intervention. Starting with the definition of the problem that is to be addressed, the theory of change represents and makes explicit how the actions that will be implemented should help to modify the existing situation, producing the desired change. Ultimately, the “theory of change” is merely a plausible, and reasonable, model of how a program should work. The theory of change can be represented graphically through the logical framework, which makes immediately visible the “ingredients” of the intervention, as well as the assumptions and threats to its operation.
Once the theory of change has been reconstructed, it is then possible to ask what one wants to know and why. It may not always make sense or be possible to meet all the knowledge needs described. The characteristics of the intervention, the number of beneficiaries, and the quantity and quality of information collected are all conditions that affect the possibility of producing robust evaluations.
The choice on the type of evaluation to be conducted must therefore be accompanied, where possible, by the design of the intervention, so that all the necessary conditions are in place to answer the questions of interest.Many of these reflections are taken up by the Guide in the chapters on monitoring and evaluation, defining the intervention rationale and structuring the logical framework.
Why do it: reap the benefits of evaluation interventions
An important consideration must concern the motivations that move the assessment.
Whatever the upstream cognitive need, evaluation should help produce knowledge useful for more informed “planning,” answering questions about the consistency of planned versus implemented activities, their replicability, and their effectiveness, including in relation to costs incurred.
It is in all cases an onerous activity, for those who fund it and for those who participate in it as observed subjects. Good then to choose carefully what to ask for and when, and to have as an objective the production of an opinion that can be used (even if negative!) for decision-making purposes.
Upcoming insights
In the following articles we will elaborate on the applicability of these three types of evaluation to European projects, their conditions of feasibility, and provide examples and experiences of evaluating some European projects.
A more general discussion of the topic can be found here, and an in-depth look at the activities and tools offered by ASVAPP can be found here.
The article reflects the terminology adopted by ASVAPP, so some discrepancies from that adopted in other sections of the Guide are possible.
